DateTime: as usual Randall Monroe is clued up

[tggzzz]

This will be appreciated by everyone that has had to explain to someone why dates and times are recursively more difficult than most people "think" (and I use that word loosely).

Don't believe me? Start by asking them how many seconds there are in a minute, how many hours there are in a day, and how many months there are in a year.  I didn't realise the relatavistic differences, and I'm not sure how I could observe them

The caption is "It's not just time zones and leap seconds. SI seconds on Earth are slower because of relativity, so there are time standards for space stuff (TCB, TGC) that use faster SI seconds than UTC/Unix time. T2 - T1 = [God doesn't know and the Devil isn't telling.]"

[nctnico]

Not just for space. Data in fiber optic connections has different speeds depending on whether the signal travels with the earth's rotation or against it. For some systems this matters enough that it needs to be compensated for.

In general I loath dealing with date & time in software but somehow I ended up involved with the highest end large scale time&frequency distribution systems.

[coppice]

I used to take recordings of sensor data recorded on aircraft, and run them through simulations of back end systems at a fraction of real time - typically 1/8th, 1/16th or 1/32nd, depending how fast we could make our simulation. So, the concept of "real time" became vague. Was real time the sensor recording domain, or our replay domain? We settled on jargon based on the movie "This Island Earth". "10 our our Earth minutes" became time on the clock in the room, while "10 minutes of real time" became 10 minutes on the aircraft. Get your jargon right, and you are good to go.

[tggzzz]

I used to take recordings of sensor data recorded on aircraft, and run them through simulations of back end systems at a fraction of real time - typically 1/8th, 1/16th or 1/32nd, depending how fast we could make our simulation. So, the concept of "real time" became vague. Was real time the sensor recording domain, or our replay domain? We settled on jargon based on the movie "This Island Earth". "10 our our Earth minutes" became time on the clock in the room, while "10 minutes of real time" became 10 minutes on the aircraft. Get your jargon right, and you are good to go.

Get your jargon right, and the next subtle "issue" will reveal itself Recursively

[tggzzz]

Not just for space. Data in fiber optic connections has different speeds depending on whether the signal travels with the earth's rotation or against it. For some systems this matters enough that it needs to be compensated for.

And for some applications the speed of light in optical fibres is too slow. Hence the High Frequency Trading mob buying up the microwave towers between Chicago and New York, to save a few milliseconds propagation delay. But that mob also encode trading business rules in FPGA hardware, to keep latency low.

For example...
In terms of niche jobs for developers in the financial services industry, it’s hard to get more niche than roles for engineers who work on ultra-ultra low latency systems at systematic hedge funds and high speed market making firms. They don’t come up often, and when they do, they require a special talent: the ability to program field-programmable gate arrays (FPGAs). Both quantitative hedge funds and high speed market making firms hire FPGA engineers. In finance, they’re found everywhere from Citadel Securities, to DE Shaw, Susquehanna International, Maven Securities and Jump Trading. Few come from a financial services background: most are drawn from chip design and hardware engineering firms. Ben Hodzic, executive director specializing in quantitative analytics at recruitment firm Selby Jennings in New York, says they can be hard to come by: there simply aren’t that many FPGA programmers to go around. “Most trading firms will consider FPGA talent from out of industry, given how hard they are to find and how niche of a skill set it is.”
https://www.efinancialcareers.com/news/2021/07/fpga-engineer-salaries-hedge-funds-market-makers

[tom66]

Not just for space. Data in fiber optic connections has different speeds depending on whether the signal travels with the earth's rotation or against it. For some systems this matters enough that it needs to be compensated for.

And for some applications the speed of light in optical fibres is too slow. Hence the High Frequency Trading mob buying up the microwave towers between Chicago and New York, to save a few milliseconds propagation delay. But that mob also encode trading business rules in FPGA hardware, to keep latency low.

I believe they're actually using ionospheric transmission now at very low frequencies.

A colleague of mine worked on a HFT trading system where the trading decision could be made on the basis of receiving the first 20 bytes out of the 200 byte packet, because that gave some better arbitrage.  Apparently they'd get it wrong now and then but it won more often than not that it was worth making the decision based on less data.

[tggzzz]

Not just for space. Data in fiber optic connections has different speeds depending on whether the signal travels with the earth's rotation or against it. For some systems this matters enough that it needs to be compensated for.

And for some applications the speed of light in optical fibres is too slow. Hence the High Frequency Trading mob buying up the microwave towers between Chicago and New York, to save a few milliseconds propagation delay. But that mob also encode trading business rules in FPGA hardware, to keep latency low.

I believe they're actually using ionospheric transmission now at very low frequencies.

Wouldn't that imply a very low data rate, and corresponding high latency? My almost negligible understanding is that HFT is based on very large numbers of trades plus cancellations or reverse trades.

A colleague of mine worked on a HFT trading system where the trading decision could be made on the basis of receiving the first 20 bytes out of the 200 byte packet, because that gave some better arbitrage.  Apparently they'd get it wrong now and then but it won more often than not that it was worth making the decision based on less data.

Given correlation with previous trade messages, I can sort of believe how that might work.

[tom66]

Wouldn't that imply a very low data rate, and corresponding high latency? My almost negligible understanding is that HFT is based on very large numbers of trades plus cancellations or reverse trades.

My understanding is it provides better overall latency because they can eliminate the repeaters required by a microwave transmission network.  Microwave transmission is only line of sight, so it requires periodic repeaters, which each add latency and cost.  Ionospheric transmission doesn't require this and also works for cross-Atlantic trades so you can beat conventional fibre HFT between London/Frankfurt stock exchanges and New York/Chicago.

The data rate is certainly low.  This Bloomberg article suggests it would be limited to individual shares or market elements like Fed rate changes.
https://archive.ph/BpMMx#selection-4967.97-4973.544

[coppice]

Wouldn't that imply a very low data rate, and corresponding high latency? My almost negligible understanding is that HFT is based on very large numbers of trades plus cancellations or reverse trades.

My understand is it provides better overall latency because they can eliminate the repeaters required by a microwave transmission network.  Microwave transmission is only line of sight, so it requires periodic repeaters, which each add latency and cost.  Ionospheric transmission doesn't require this and also works for cross-Atlantic trades so you can beat conventional fibre HFT between London/Frankfurt stock exchanges and New York/Chicago.

The data rate is certainly low.  This Bloomberg article suggests it would be limited to individual shares or market elements like Fed rate changes.
https://archive.ph/BpMMx#selection-4967.97-4973.544

Let's see. Balloons at each end to increase the horizon distance. Reflectors on each balloon, to avoid slow transmission up a cable or repeater delays. Chicago to New York is 800 miles. I think that means the balloons need to be about 30km up. Now we just need to decide how to stabilise their positions and we have a plan.

[coppice]

Wouldn't that imply a very low data rate, and corresponding high latency? My almost negligible understanding is that HFT is based on very large numbers of trades plus cancellations or reverse trades.

My understanding is it provides better overall latency because they can eliminate the repeaters required by a microwave transmission network.  Microwave transmission is only line of sight, so it requires periodic repeaters, which each add latency and cost.  Ionospheric transmission doesn't require this and also works for cross-Atlantic trades so you can beat conventional fibre HFT between London/Frankfurt stock exchanges and New York/Chicago.

The data rate is certainly low.  This Bloomberg article suggests it would be limited to individual shares or market elements like Fed rate changes.
https://archive.ph/BpMMx#selection-4967.97-4973.544

If both ends of the link pre-agree on a list of possible actions, they might only need to exchange a couple of bytes of data + some error checking. However, if the data rate is really low even such small messages might represent a latency problem.

[themadhippy]

Let's see. Balloons

so it wasnt a weather balloon causing problems earlier this year,it was a chinese stock trader .

[Nominal Animal]

Time is not something we can measure directly, only elapsed time; and in different locations, between any two events, the elapsed time varies.  In some extreme cases, even the observed order of cause and effect can be reversed.

There are different classes of problems stemming from this.  The nastiest one, I believe, is synchronization: consider GPS, or communication with space probes.

There are several clocks, or timekeeping bases, we use here on Earth.  The most common is UTC or Coordinated Universal Time, based on TAI or International Atomic Time, with leap seconds added or subtracted as needed.  (Thus, one minute can be 59 to 61 seconds long, even though almost all minutes are 60 seconds long.)

Unix time as used on most computers is based on UTC.

It is notable that TAI is the average of over 450 atomic clocks all over the Earth, so it does not represent the elapsed time on any point, but the average elapsed time on the surface of Earth since a preset epoch, that epoch itself purely notional (not physically verifiable or measurable at all).

To convert from UTC to local time, you need two databases: one is the leap second adjustment table (identifying when leap seconds were added or removed), and the other is the local time database including daylight saving times.  You need the leap second adjustment table even for counting the exact number of seconds between two UTC date-times.

All this means is that our timekeeping is not, and cannot be exact.

Realizing this helps, because then you start thinking about what kind of precision is useful.  And then, the timekeeping related problems start falling into different categories (depending on how important they are to the problem at hand), becoming just another numerical/physical (as opposed to algorithmic/theoretic) problem to be solved.

In numerical physical simulations with discrete time steps – involving anything from individual atoms, to stars and galaxies –, one key "trick" is to dynamically adjust the time step size to the scale at which events can be represented at useful precision.  (It makes it MUCH harder for us humans to observe the time evolution of the system, though; our brains cannot really deal with time "speeding up" or "slowing down".)

In many cases from programming to humans waking up from sleep, the exact moment something is scheduled to happen in the future is less important than that moment being suitable.  (You want to wake up from non-REM sleep, because waking up from REM sleep leaves you groggy.  You don't want your computer to do housekeeping stuff like checking for updates while you're working or watching media full-screen, because it leads to jerkiness or added latencies.)

All this leads to moving away from "do X at time Y", into "do X after time Y1 but before time Y2": moving away from specific moments or points in time, into time intervals.

When you start really working with different aspects of time, it starts changing the way you think about it.  (In a good way, I believe.)

[tggzzz]

Time is not something we can measure directly, only elapsed time; and in different locations, between any two events, the elapsed time varies.  In some extreme cases, even the observed order of cause and effect can be reversed.

There are different classes of problems stemming from this.  The nastiest one, I believe, is synchronization: consider GPS, or communication with space probes.

In a distributed system there can be no common concept of time, and the best you can achieve is partial ordering.
Leslie Lamport's seminal papers had something to say about that back in 1978 http://research.microsoft.com/users/lamport/pubs/time-clocks.pdf
He won a Turing Award for that! https://amturing.acm.org/award_winners/lamport_1205376.cfm

As for dates, realise that (amongst many many wierdnesses):

• in the UK the date 5th September 1752 simply does not exist[1]
• years have not always started on January 1st, even in Europe. 1751 in the UK (and the eastern bits of what has become the USA) was only 282 days long

[1] I first came across this when I was about 10yo. I read a story where some kids were able to prove a document was a forgery because it was dated 5th September 1752.

[bill_c]

Tom Scott about date and time.

[Nominal Animal]

Time is not something we can measure directly, only elapsed time; and in different locations, between any two events, the elapsed time varies.  In some extreme cases, even the observed order of cause and effect can be reversed.

There are different classes of problems stemming from this.  The nastiest one, I believe, is synchronization: consider GPS, or communication with space probes.

In a distributed system there can be no common concept of time, and the best you can achieve is partial ordering.
Leslie Lamport's seminal papers had something to say about that back in 1978 http://research.microsoft.com/users/lamport/pubs/time-clocks.pdf
He won a Turing Award for that! https://amturing.acm.org/award_winners/lamport_1205376.cfm

Exactly!

Authentication on the interwebs is a perfect example.  From the server point of view, there is no ordering to the requests a specific client makes, and they can even be interspersed (as in more than one concurrent page load, with additional requests loading associated files, all at the same time).  You cannot update the authentication information on each request, because each response can generate more than one request, and there is still no ordering.  You can keep all information on the server, so that the authentication cookie is just a large random number with zero information in it, but that means servers need to share the same active authentication database.  Instead, some kind of time-dependent field is used in the cookie.

Because there is no common concept of time, regardless of the validity duration of the time-dependent field (say, if it is in minutes, hours, or days, or anything in between), the server side has to be ready to accept the previous and the next value of said field –– either both, or previous for the first half of the validity interval, and next for the latter half of the validity interval.

There is no "today"; one must consider "yesterday", "today", and "tomorrow" concurrently.

Then, you add to the fact that sometimes, it is possible for the client IP (IPv4 especially, much less so for IPv6) address to change during an authorized session.  How do you make sure it is not just an authentication cookie hijacking?  If you include the client address or hash of the client address in the authentication cookie, you can detect when that (IP address change, or cookie replay/hijack attack) happens.  Time, or order of events is not something you can really rely on, but a backup cookie that has not been used since authentication (limited by cookie path) can be of use.  (If an attacker has the entire client browser state, they're in full control of the session anyway.)

Ones normal sequential view of time just doesn't work here; one must really understand intervals and how event ordering can vary.

And, whenever dealing with secure stuff, how event ordering can be exploited.  CVE-1999-0035 is a good example of that: the good ol' ftpd SIGINT bug, which allowed superuser file access to a server just by sending Ctrl+C at the right moment, because ftpd developers didn't realize what might happen if the signal was delivered at just the right moment.

[tggzzz]

I've occasionally wondered how fleabay manages to be "fair" and "accurate" at the end of auctions, especially with sniping from different continents. I've no real way of telling whether it actually does possess those properties.

The HFT mob has timescales a thousand times worse, and they have the inclination and money to be sure

[coppice]

I've occasionally wondered how fleabay manages to be "fair" and "accurate" at the end of auctions, especially with sniping from different continents. I've no real way of telling whether it actually does possess those properties.

The HFT mob has timescales a thousand times worse, and they have the inclination and money to be sure

"fair" and "accurate", or just "fairly accurate"? I expect fairly accurate is good enough. Most of the latencies are beyond their control, so they really can't do better. It be happy if e-bay just sorted out their broken language selection system.

[nctnico]

I've occasionally wondered how fleabay manages to be "fair" and "accurate" at the end of auctions, especially with sniping from different continents. I've no real way of telling whether it actually does possess those properties.

Is there a rule for auction sites? My guess is that Ebay's software just doesn't care and rounds timestamps to the nearest second. It simply is not a problem.

[tggzzz]

I've occasionally wondered how fleabay manages to be "fair" and "accurate" at the end of auctions, especially with sniping from different continents. I've no real way of telling whether it actually does possess those properties.

Is there a rule for auction sites? My guess is that Ebay's software just doesn't care and rounds timestamps to the nearest second. It simply is not a problem.

I haven't noticed a problem, which is a very weak statement.
I haven't seen reports of a problem, which is a slightly less weak statement.

[nctnico]

I've occasionally wondered how fleabay manages to be "fair" and "accurate" at the end of auctions, especially with sniping from different continents. I've no real way of telling whether it actually does possess those properties.

Is there a rule for auction sites? My guess is that Ebay's software just doesn't care and rounds timestamps to the nearest second. It simply is not a problem.

I haven't noticed a problem, which is a very weak statement.
I haven't seen reports of a problem, which is a slightly less weak statement.

Once again you are trying to create a problem which just isn't there. First come up with a legal requirement that states how auctions are supposed to be held timing wise. If that exists, you have a framework you can build a problem definition upon. But if that isn't there, there simply is no problem because there are no requirements. IOW: a problem can not exist without requirements.

If you go to Ebay's website and read about auction timing, you'll see they specify the time an auction runs in units of 1 minute. Basically it means whatever works best for Ebay goes.

[thm_w]

It doesn't really matter with Ebay, the person who bids the highest before the time ends will win. The point of sniping is another bidder won't have time to go back to their computer and type in a new higher number, so anything 5-10s before the end of the auction should be plenty. Its almost better if its on the higher end as if two people bid $100 the earlier bid wins.

Sniping services will let you choose a buffer, and recommend an absolute minimum of 3 seconds for network/processing delays. Good services will also submit from multiple servers at the same time. Which tells me ebays server relies on its own internal timekeeping and doesn't care about what timestamp you have on your packet.

[tggzzz]

I've occasionally wondered how fleabay manages to be "fair" and "accurate" at the end of auctions, especially with sniping from different continents. I've no real way of telling whether it actually does possess those properties.

Is there a rule for auction sites? My guess is that Ebay's software just doesn't care and rounds timestamps to the nearest second. It simply is not a problem.

I haven't noticed a problem, which is a very weak statement.
I haven't seen reports of a problem, which is a slightly less weak statement.

Once again you are trying to create a problem which just isn't there. First come up with a legal requirement that states how auctions are supposed to be held timing wise. If that exists, you have a framework you can build a problem definition upon. But if that isn't there, there simply is no problem because there are no requirements. IOW: a problem can not exist without requirements.

If you go to Ebay's website and read about auction timing, you'll see they specify the time an auction runs in units of 1 minute. Basically it means whatever works best for Ebay goes.

Once again you are over-interpreting individual words, thus seeing trees and missing the wood under discission.

Even trivial experiments using a single browser without using sniping services demonstrate that the granularity of specifying an auction's end is irrelevant to the points being discussed.

[tggzzz]

It doesn't really matter with Ebay, the person who bids the highest before the time ends will win. The point of sniping is another bidder won't have time to go back to their computer and type in a new higher number, so anything 5-10s before the end of the auction should be plenty. Its almost better if its on the higher end as if two people bid $100 the earlier bid wins.

That's one reason not to bid a round number.

Sniping services will let you choose a buffer, and recommend an absolute minimum of 3 seconds for network/processing delays. Good services will also submit from multiple servers at the same time. Which tells me ebays server relies on its own internal timekeeping and doesn't care about what timestamp you have on your packet.

That raises the question of how multiple competing sniping services interact.
 
I haven't used a sniping service, so am unaware of the available parameters. Using multiple servers is an obvious optimisation, but I'm not clear about the precise benefits.

Using multiple browsers and/or windows pointed at the same auction can show "seconds to go" values differing by up to a second (?or slightly more?). It is unclear to me what happens if you somehow manage to submit a bid when one window shows 0s but another shows 1s.

It is also unclear to me what correlation there is or isn't between the "seconds to go" values on different computers, and if/how eBay uses that information.

[thm_w]

I haven't used a sniping service, so am unaware of the available parameters. Using multiple servers is an obvious optimisation, but I'm not clear about the precise benefits.

The benefit is if one server can't connect, power dies, packet gets dropped, packet gets hung up, whatever, then you'll still have another packet that can go through.

Using multiple browsers and/or windows pointed at the same auction can show "seconds to go" values differing by up to a second (?or slightly more?). It is unclear to me what happens if you somehow manage to submit a bid when one window shows 0s but another shows 1s.

It is also unclear to me what correlation there is or isn't between the "seconds to go" values on different computers, and if/how eBay uses that information.

If the PC time is off and you submit the bid late, you would just get an error message.
After googling I see a lot of complaints that ebay leaves off the ending time in seconds in some places, so the auction might end at :00 or :59 which confuses them. Also there seems to be a recent issue that if your PC time is too fast, it can cause bidding submit problems (so your browser will think the auction has ended when it hasn't).

[SiliconWizard]

It's all relative.