BGB

On 2/12/2012 7:27 AM, Arne Vajhøj wrote:
> On 2/12/2012 1:03 AM, BGB wrote:
>> On 2/11/2012 1:18 PM, Arne Vajhøj wrote:
>>> On 2/11/2012 3:06 PM, Lew wrote:
>>>> XML is just fine for just about every purpose to which it's put.
>>>> That's why it's popular now. People who cavil about "bandwidth" and
>>>> "10 Hz network messages" are tossing us red-herring sashimi. You
>>>> aren't going to get 10+ Hz message exchanges over the WAN. For
>>>> realistic message rates, XML suits beautifully. I speak from
>>>> experience with many, many projects that used every conceivable
>>>> message format from binary to CSV to custom to XML to protocol buffers
>>>> to JSON, and XML has distinct advantages. Its purported disadvantages
>>>> of bulk and bandwidth turn out to be non-issues in practice. Really.
>>>> That's real.
>>>
>>> ????
>>>
>>> The industry standard for smartphone apps and AJAX web apps
>>> are JSON because the bandwidth actually matters.
>>
>> yeah.
>
> But please note that you do not invent your own JSON parser
> either - you use something already done. In Java there are json.org,
> gson etc..
>

in Java, yes, one may use the libraries.

on the C end, one may choose to throw one together, or use a JavaScript
VM if one is available, ...

it all depends.


>> and, in fact 10Hz asynchronous throughput is possible... (this is
>> typically how most online gaming works).
>>
>> one can't reasonably get 10Hz synchronous (IOW: request/response), due
>> to ping times, but this is a different matter.
>>
>> I am not claiming one can do request/response at 10Hz, as this probably
>> would be impossible over a WAN (at reasonable ping times).
>>
>> online gaming generally isn't based on request/response though, so the
>> problems of not being able to get responses at this rate don't really
>> matter too much (the user may still notice the results of bad ping
>> times, namely stuff being out of place or seemingly teleporting around
>> or similar, but this is a different issue).
>>
>> TCP has a throughput limit due to ping, where assuming unlimited
>> bandwidth but a 200ms ping, one will still be limited to somewhere
>> around 320kB/s or so (assuming a 64kB window).
>>
>> however, as is, there are typically bandwidth constraints as well (but,
>> they are a little more subject to fudging).
>>
>> the main goal is mostly to send everything at a bandwidth low enough
>> that the connection doesn't risk getting backlogged (depends on internet
>> connection, but 16-32 kB/s seems reasonably safe, as internet radio
>> often tends to operate in this range, say, 192kbps is 24kB/s).
>>
>> stalls are also a potential risk as well.
>
> I can not follow you way of thinking. With multiple interactions
> in parallel there are no strict correlation between latency and
> throughput.
>

there is a rough correlation though.


for the part about TCP, this was related to how TCP worked (in its
traditional form), namely the existence of a 64kB maximum window size.

apparently, this is out of date, as there is a feature known as
RFC-1323, which is enabled by default on Windows Vista and newer, which
allows a larger TCP window.

http://tools.ietf.org/html/rfc1323


for the part about moderating kB/s, this has a lot more to do with a
users' internet connection.

say, hypothetically, a user has dial-up.

now, what if the data being sent does not fit over dial-up (one is
trying to send 10kB/s, but a 56k modem can only handle ~6.5kB/s or so)?
well, then, the connection will backlog (the connection will send at the
rate it can send, and anything else will have to wait).

similar limits may exist over the internet, but in a less direct form:
consider, the internet is prone to occasionally drop a packet here or there.

so, stream is going over the internet, and a (single) packet drops, what
happens:
well, all the data up to the dropped packet reaches the other end, the
other end may send a packet back indicating the point recieved;
the sender will start resending data from that point;
the reciever will start transmitting again.

this results in essentially a ping-time delay in which no data can be sent.

if the sender is sending messages at a fixed rate, what happens?
well then, the messages will pile up, waiting to be sent;
after transmission resumes, several updates worth of data need to be sent;
if all of the updates fit within the bandwidth of the connection
(end-to-end), then there is may be no obvious stall (updates can all be
sent at full speed);
if the enough data back-logs so as to exceed the bandwidth available,
then it has to wait to be sent, and if the sender just keeps naively
sending updates, then essentially one gets a stall (and the data being
received by the receiver will start becoming progressively more
out-of-date).


these properties can be observed with things like internet radio and
video streaming (if the connection is fast enough, playback happens in
real-time without obvious stalls or re-buffering, even though the rate
at which the data comes over the internet is often very irregular).

similar also applies to internet telephony as well.


if one tries to operate within a fixed-bandwidth window, similar to
internet radio, most minor stalls can be glossed over (this limit being
a bit lower than the end-to-end transfer rate of the connection). going
lower is better, since the lower one goes, the more room there for error
there is.

the main issue is, namely, that the data being sent has to be able to
fit within these bandwidth limits (hence, why data compression is highly
desirable in this case).


an online game basically amounts to a bidirectional stream between the
client and server, with the server sending out a stream of updates
(typically, everything going on in the immediate view of the client),
and the client sends a stream of their attempted actions (in response to
what they see on screen).

if everything is working well, then the delays and irregularities of
their internet connection is mostly hidden, and to them it all seems
like they are interacting with the world in real-time (usually there is
a lot of trickery here as well, mostly based around linear extrapolation
and so on).

side note: each end may transmit time-stamps as part of their updates,
and the other end may transmit the last-received timestamps, partly so
that the timing delays can be estimated and partially compensated for.


another (similar concept) for players playing games is the concept of
"leading", where a person will take aim at a moving enemy, estimate the
speed of the projectile and where the enemy will be at the time, and aim
and fire at that location instead (then the enemy will essentially "run
into" the traveling projectile). note that if a player always aims at
where the enemy is "right now", very often they will miss (as by the
time the projectile reaches the destination, the enemy has already moved
out of the way).

so, the game does similar in an attempt to hide the "travel time" that
is the internet.


or such...

Arne Vajhøj

On 2/12/2012 3:33 PM, BGB wrote:
> On 2/12/2012 7:27 AM, Arne Vajhøj wrote:
>> On 2/12/2012 1:03 AM, BGB wrote:
>>> On 2/11/2012 1:18 PM, Arne Vajhøj wrote:
>>>> On 2/11/2012 3:06 PM, Lew wrote:
>>>>> XML is just fine for just about every purpose to which it's put.
>>>>> That's why it's popular now. People who cavil about "bandwidth" and
>>>>> "10 Hz network messages" are tossing us red-herring sashimi. You
>>>>> aren't going to get 10+ Hz message exchanges over the WAN. For
>>>>> realistic message rates, XML suits beautifully. I speak from
>>>>> experience with many, many projects that used every conceivable
>>>>> message format from binary to CSV to custom to XML to protocol buffers
>>>>> to JSON, and XML has distinct advantages. Its purported disadvantages
>>>>> of bulk and bandwidth turn out to be non-issues in practice. Really.
>>>>> That's real.
>>>>
>>>> ????
>>>>
>>>> The industry standard for smartphone apps and AJAX web apps
>>>> are JSON because the bandwidth actually matters.
>>>
>>> yeah.
>>
>> But please note that you do not invent your own JSON parser
>> either - you use something already done. In Java there are json.org,
>> gson etc..
>>
>
> in Java, yes, one may use the libraries.
>
> on the C end, one may choose to throw one together, or use a JavaScript
> VM if one is available, ...

There are plenty of libs for C as well.

30 seconds of googling found:

http://www.digip.org/jansson/
http://live.gnome.org/JsonGlib
http://sourceforge.net/projects/mjson/
http://oss.metaparadigm.com/json-c/

Arne

BGB

On 2/12/2012 1:50 PM, Arne Vajhøj wrote:
> On 2/12/2012 3:33 PM, BGB wrote:
>> On 2/12/2012 7:27 AM, Arne Vajhøj wrote:
>>> On 2/12/2012 1:03 AM, BGB wrote:
>>>> On 2/11/2012 1:18 PM, Arne Vajhøj wrote:
>>>>> On 2/11/2012 3:06 PM, Lew wrote:
>>>>>> XML is just fine for just about every purpose to which it's put.
>>>>>> That's why it's popular now. People who cavil about "bandwidth" and
>>>>>> "10 Hz network messages" are tossing us red-herring sashimi. You
>>>>>> aren't going to get 10+ Hz message exchanges over the WAN. For
>>>>>> realistic message rates, XML suits beautifully. I speak from
>>>>>> experience with many, many projects that used every conceivable
>>>>>> message format from binary to CSV to custom to XML to protocol
>>>>>> buffers
>>>>>> to JSON, and XML has distinct advantages. Its purported disadvantages
>>>>>> of bulk and bandwidth turn out to be non-issues in practice. Really.
>>>>>> That's real.
>>>>>
>>>>> ????
>>>>>
>>>>> The industry standard for smartphone apps and AJAX web apps
>>>>> are JSON because the bandwidth actually matters.
>>>>
>>>> yeah.
>>>
>>> But please note that you do not invent your own JSON parser
>>> either - you use something already done. In Java there are json.org,
>>> gson etc..
>>>
>>
>> in Java, yes, one may use the libraries.
>>
>> on the C end, one may choose to throw one together, or use a JavaScript
>> VM if one is available, ...
>
> There are plenty of libs for C as well.
>
> 30 seconds of googling found:
>
> http://www.digip.org/jansson/
> http://live.gnome.org/JsonGlib
> http://sourceforge.net/projects/mjson/
> http://oss.metaparadigm.com/json-c/
>

fair enough.

I might look into it, although personally I don't use JSON for this at
the moment, and if it were needed in my-case, as-is my script VM can
parse JSON (given the language used is a superset of JavaScript
anyways), although this is potentially a less efficient strategy than a
dedicated parser.

say, in my case, it would be a tradeoff between either: using someone
else's library, passing the JSON through "eval", or spit out some logic
to make the VM parse the JSON directly (probably just copy/paste/edit
some of the existing parser code). then one could wonder secondarily:
what form would the JSON be parsed into? "there is a library for that"
is not always necessarily the least-effort option.

either way, one still might want to save more bytes, say, by running it
through deflate or similar. if one wants libraries, Java has it built
in, and in C-land there is zlib.

in my case, I also have a deflate codec which is stored as a single big
source file, mostly as this makes it a little more convenient (in
several ways) than using zlib. JPEG was similar, as originally I used
libjpeg, but reimplemented JPEG as a "single big source file" to be a
generally more convenient option (copy/paste the source file and go).

I don't claim a person might "always" want to do this, but as I see it,
it is still a potentially valid option. one could maybe go further, and
put Deflate, JPEG, PNG, and several other formats, all in a single big
file, at the drawback of the file becoming overly large.

similarly, the above probably would be fairly pointless in Java, both
because this stuff exists in the standard library, and also because this
would also result in a single giant class as well.


the cheapest option for one person may well turn out to be a more
expensive option for another person, say if they one of the options
amounts to "implement the functionality from the ground up" rather than
"copy-paste a few bits from over there and hack something together", or
even maybe just "add a few lines in a function over here and add a new
function or method over there which redirects the call to the first
function".

all of this stuff can be fairly relative, and there are rarely "cut and
dry" answers to problems.



as for the delay issue, found this article on part of the topic:
http://en.wikipedia.org/wiki/Lag_%28online_gaming%29

also maybe relevant:
http://en.wikipedia.org/wiki/Internet_streaming

BGB

On 2/12/2012 12:52 PM, Joshua Cranmer wrote:
> On 2/12/2012 2:36 AM, BGB wrote:
>> safely, one can use, on Windows:
>> the ANSI C runtime (more or less C89/C90);
>> any Win32 API provided stuff (Winsock, GDI, OpenGL, ...);
>> ...
>
> I call BS on this, having worked on a major open-source project that
> works on the major platforms of Windows, Mac OS X, and Linux (and also
> Android, and I think Solaris and *BSD are still reasonably
> well-supported, although the Haiku and OS/2 ports are now thoroughly
> dead). What libraries does this? A small list:
> * libpng
> * libjpg
> * libogg + related
> * cairo
> * thebes
> * sqlite
> * freetype
> * libbz2
> * libjar
> * zlib
>
> And all of these are still used on Windows; there are even more that are
> used only on Linux or Mac OS X.
>
> Which application is this? Mozilla.
>
> It's not that hard to use other libraries on Windows.
>

yes, but it is also worth noting that Mozilla does the whole "Mozilla
build" thingy on Windows, and are essentially they are bundling many of
the needed libraries and tools with the application as a part of the
build system.


as noted:
it is not saying that one *can't* use 3rd party libraries, but one may
need to make special provisions for them.

Mozilla does this.

not everyone may want to do this, as it is a reasonably heavy-weight
solution to the problem (but still better than "hey random person, go
download and build all of these libraries yourself", which is what some
applications have gone and done).

Martin Gregorie

On Sun, 12 Feb 2012 11:16:03 -0500, Arne Vajhøj wrote:

>
> Doesn't getopt exist in some GNU lib that you can get for all platforms?
>
Pass. I know it was missing from Borland C on DOS and Windows and wasn't
published for OS-9. I ended up extending a PD version that was originally
published in the '68 MicroJournal for OS-9 and porting it to the other
two platforms.

I had a good (to me) reason for doing that. At the time I was more
familiar with OS-9 than Windows or Unix (Linux didn't exist at the time)
and I had got used to the OS-9 command line parser's ability to handle a
mix of options and arguments in any order rather than the straitjacket of
Unix's rigid options before arguments rule. My extension basically just
added the -x=value notation (also used by OS-9) to the standard Unix -
xvalue and -x value notation. I've since rewritten it for Java, adding
long option names (--xxxx and --xxxx=val) - something I've not gotten
round to adding to the C version.


--
martin@ | Martin Gregorie
gregorie. | Es***, UK
org |