Problem using BFStats to generate server stats

Black Mamba · 2018-04-25 05:41:13

I am having issues using BFStats, which is a program which was made by Mario Kohler years ago to convert CSV spreadsheets generated by BFSM (statistics.csv) to HTML pages that can be uploaded via FTP to a website.

I used the following tutorial to try and generate the HTML pages: http://elitesniperserver.com/media/tuto … e.asp?id=9

First, I downloaded and installed the latest Java SDK + Runtime Kit as pointed out by the tutorial by visting this website: http://www.oracle.com/technetwork/java/ … 33151.html

I then downloaded the "Java SE Development Kit 8u172" for Windows x64 (jdk-8u171-windows-x64.exe). After that I downloaded and installed the BFStats.zip and extracted it to a folder on my Desktop called "Stats". Here is the file directory inside this Stats folder:

The statistics.csv file was generated by the BFRM v2.01 using the following settings (see the image below). I don't know if enabling Event log compression is the issue here, but I doubt it.

Now the tricky part. After having edited the "run.cmd" file found inside the "bin" folder with a text editor to point to the path where I have stored the statistics.csv file, I run the file and it gives me the following errors:

Basically it's saying that the statistics.csv file is empty even though it is not (red rectangle). I have opened it many times and can see all the games being logged there, with the player names, their scores, pings, deaths, IPs, etc. Running the file with Administrator Rights also does nothing and does not generate the HTML files inside the "www" folder.

I am using Windows 10 Home 64-bit. Has anyone else had problems with this program to generate HTML files from the CSV file?

Last edited by Black Mamba (2018-04-27 17:05:52)

Trench · 2018-04-26 15:24:03

The "Java.lang.NullPointerException" is saying that something went critically wrong for the BFStats Java code when attempting to parse the line from the .CSV file. And so it's ignoring the line as potentially "invalid." But this happens during the parsing of every single line. And so with 100% of lines ignored, BFStats declared "the file is empty" because there were no valid lines processed.

No, I've never used BFStats and don't know what it will take to solve this. My guesses would be that possibly the Java runtime itself has long-since closed some security hole or bad coding that this application used to get away with. Or that the data being generated into the .CSV doesn't contain data that BFStats always assumed would be there, and doesn't parse correctly without.

The BFStats program appears to have been created before BFServerManager 2.01 was out, so possibly its some difference in the format of the .CSV logging as compared to 2.0 or similar. Maybe if you can temporarily run with 2.0 and generate a .CSV log from that, to compare against your 2.01 logs. If there is a specific difference, even though you can't fix the BFStats program, maybe you can manually or via macros edit your 2.01 log files to be consistent with the 2.0 format before presenting them to BFStats.

I don't decompile or write Java code, so that's as far as I can reasonably guess.

bud · 2018-04-27 05:08:07

Its supposed to put the result in a database, like MySQL

Trench · 2018-04-27 07:58:35

bud wrote:

Its supposed to put the result in a database, like MySQL

Does that actually apply to this particular tool? Because the readme file seems to describe exactly what I would expect, in that it simply converts the CSV data into an HTML page representing that CSV data. i.e. The CSV file "is the database" in this tool's case.

If creating or configuring a formal SQL database was involved, it seems like that would need "at least a passing mention" in the readme. But again, I haven't used this specific tool. I just know I wouldn't need a database in order to do what this tool says its going to do.

-Trench

bud · 2018-04-28 05:49:03

Oh, you are right! I was thinking of another stats software that was popular years ago. my fault

edit: i get the same error here

Last edited by bud (2018-04-28 06:02:32)

Black Mamba · 2018-04-28 06:29:08

Well that sucks... any other way to generate stats?

(BB)DinkW · 2018-04-28 10:49:50

If you are just trying to make the CSV file into HTML you could try this

http://www.convertcsv.com/csv-to-html.htm

Trench · 2018-04-28 16:14:05

Black Mamba wrote:

Well that sucks... any other way to generate stats?

I haven't run any scores/kills/deaths stats to recommend anything first-hand.

I see https://selectbf.sourceforge.io/ and https://github.com/SiphonSquirrel/jepperscore out there, but they don't appear to be for the faint of heart technical-wise. (I believe selectbf must have been the one bud was referring to earlier.) These systems are also for collating BF1942's own more extensive event/XML logging and not the BFServerManager CSV logging.

If you were happy with the data that was promised from just the CSV logging (just maps, players, kills, deaths), the only way I know to collate and put that into human-readable form like BFStats was going to do would be to write something yourself.

Black Mamba · 2018-04-28 19:59:55

Trench wrote:

Black Mamba wrote:
Well that sucks... any other way to generate stats?
I haven't run any scores/kills/deaths stats to recommend anything first-hand.
I see https://selectbf.sourceforge.io/ and https://github.com/SiphonSquirrel/jepperscore out there, but they don't appear to be for the faint of heart technical-wise. (I believe selectbf must have been the one bud was referring to earlier.) These systems are also for collating BF1942's own more extensive event/XML logging and not the BFServerManager CSV logging.
If you were happy with the data that was promised from just the CSV logging (just maps, players, kills, deaths), the only way I know to collate and put that into human-readable form like BFStats was going to do would be to write something yourself.

I guess I'll just use an Excel spreadsheet to edit stats manually...

bud · 2018-04-29 03:55:49

I was curious so i tried it on windows xp with an old java version (2002) and it still gets the same fault message. The thing is that i have a faint memory of using this software before, or something very similiar.

edit: sometimes flashbacks can be a good thing, i realized what the difference was between then and now, it seems to be some difference in the log file format between windows vs linux. Its not something one can see at a glance tho.

Found an old windows generated csv file and tried again.
_
http://37.187.19.136/stats/index.html

Last edited by bud (2018-04-29 04:54:34)

Trench · 2018-04-29 23:31:49

bud wrote:

Found an old windows generated csv file and tried again.
http://37.187.19.136/stats/index.html

Great find. Perhaps it's just line endings then, if the difference isn't visually obvious? i.e. CR/LF versus LF-only.

If so, then simply loading up the CSV file in an editor like Notepad++ and using Edit | EOL Conversion to change the line endings might let Mamba run his existing log files through the tool. Specifically, if this is the issue, we're expecting that loading the file into Notepad++ will show down in the bottom status bar that the current line endings are "Linux (LF)", and from the Edit | EOL Conversion menu you would select "Windows (CR LF)" and then save the file with the updated line endings.

EDIT: I've only just now even turned on CSV logging on our Windows server, and never had it running on the Linux servers. The Windows log does confirm being CR/LF, as typically expected for Windows platform.

Last edited by Trench (2018-04-29 23:35:48)

bud · 2018-05-01 01:23:29

Nice! I wouldnt have thought of that. The linux file says Unix(LF), ill try convert it.

edit: gnarf! .. must be something more i guess, ill see if i can figure out what it is, i kinda like it when it works, small files, quick loading, no database to think about.
-

-
About the red underlining, first time i´ve seen three "unknown" like that, its no ip and no key (last one is punkbuster)
-

Last edited by bud (2018-05-01 01:41:52)

Trench · 2018-05-01 18:35:09

Hmm. Line endings certainly would have fit with "it fails on every single line, such that by the end it thinks there were a total of zero valid lines." So if it's not that, then there "must" be some more obvious difference in the data, such as quotes used in one where quotes weren't used in the other; a different number of total columns; or some other difference that "is true for every single line."

If you wanted to PM your Linux versus Windows examples, I'd be interested to try and identify the difference too. If the data in the examples you have isn't something you want to disseminate, that's understandable as well. It would need to be as an attachment or something else that preserves the binary format the files, so maybe admin at ea117.com is the better way to send, if you do.

Yeah, since he had ping and score stats, nothing seems logical about why that one line would have an unknown IP or unknown key hash. In the short time I've been running CSV logging, there have already been a couple times where at the end of the round it simply says "unable to save CSV stats" instead of appending them as expected. So could just be buggy.

bud · 2018-05-02 01:50:57

Yeah this is a weird one alright. I sent you a pm so you can take a closer look at those files.

Trench · 2018-05-04 04:08:35

bud wrote:

Yeah this is a weird one alright. I sent you a pm so you can take a closer look at those files.

Two conclusions:

BFStats can't parse a date from 2010 or later.

If BFStats can't parse the first date in the file, the entire file is declared unparseable. (Presumably because he wants to compare each later entry against the time stamp(s) collected, but the initial time stamp collected was invalid. So this perpetually fails for every additional line parsed from the file.)

A current three-day statistics.csv file from my own Windows-based server also fails to parse with the same "file is empty, cannot generate any output" outcome. So was able to rule out "from a Windows server" as constant for success too. After seeing java.lang.NumberFormatException errors being thrown in addition to the java.lang.NullPointerException error, the focus shifted towards all the various numbers being logged, and the difference between the successful and failing statistics.csv files.

Changing the first time stamp from the 2018 log file to replace the "1" with a "0" so that the date is 2008, and leaving the rest of the file untouched with still-2018 time stamps, and my Windows server's log now parsed and generates stats. However, the generated stats are incomplete, and the java.lang.NumberFormatException errors continue to be thrown for the remaining 2018 date stamps in the statistics.csv file. If you search and replace all the 2018 dates with 2008 (or anything under 2010), then zero errors are reported when running BFStats and complete stats are generated.

You should be able to confirm this with the failing Linux log example you had sent. The first time stamp was 2016, and fails to parse the entire file. Change that time stamp to 2010, and confirm BFStats still fails to parse the entire file. But change that first time stamp to 2009, leaving the rest of the file untouched, and now the file parses and generates output. But the stats aren't complete. Replace all "201x" time stamps with "200x" (i.e. 2016, 2017 and 2018 become 2006, 2007 and 2008), and you'll see no errors reported, and the HTML pages generated will have a huge number of additional stats compared to when you only fixed the first line.

EDIT: Missed confirming that the Windows example you sent "parses successfully", because the first date in that file happened to already be 2006. The stats generated are incomplete, however, since the remaining 2018 dates all throw java.lang.NumberFormatException errors.

-Trench

Last edited by Trench (2018-05-04 04:36:58)

Black Mamba · 2018-05-04 04:38:31

Trench wrote:

bud wrote:
Yeah this is a weird one alright. I sent you a pm so you can take a closer look at those files.
Two conclusions:
BFStats can't parse a date from 2010 or later.
If BFStats can't parse the first date in the file, the entire file is declared unparseable. (Presumably because he wants to compare each later entry against the time stamp(s) collected, but the initial time stamp collected was invalid. So this perpetually fails for every additional line parsed from the file.)
A current three-day statistics.csv file from my own Windows-based server also fails to parse with the same "file is empty, cannot generate any output" outcome. So was able to rule out "from a Windows server" as constant for success too. After seeing java.lang.NumberFormatException errors being thrown in addition to the java.lang.NullPointerException error, the focus shifted towards all the various numbers being logged, and the difference between the successful and failing statistics.csv files.
Changing the first time stamp from the 2018 log file to replace the "1" with a "0" so that the date is 2008, and leaving the rest of the file untouched with still-2018 time stamps, and my Windows server's log now parsed and generates stats. However, the generated stats are incomplete, and the java.lang.NumberFormatException errors continue to be thrown for the remaining 2018 date stamps in the statistics.csv file. If you search and replace all the 2018 dates with 2008 (or anything under 2010), then zero errors are reported when running BFStats and complete stats are generated.
You should be able to confirm this with the failing Linux log example you had sent. The first time stamp was 2016, and fails to parse the entire file. Change that time stamp to 2010, and confirm BFStats still fails to parse the entire file. But change that first time stamp to 2009, leaving the rest of the file untouched, and now the file parses and generates output. But the stats aren't complete. Replace all "201x" time stamps with "200x" (i.e. 2016, 2017 and 2018 become 2006, 2007 and 2008), and you'll see no errors reported, and the HTML pages generated will have a huge number of additional stats compared to when you only fixed the first line.
EDIT: Missed confirming that the Windows example you sent "parses successfully", because the first date in that file happened to already be 2006. The stats generated are incomplete, however, since the remaining 2018 dates all throw java.lang.NumberFormatException errors.
-Trench

I tried the method you described on your second paragraph and no errors appear. The HTML files are generated in the correct folder and everything seems to work.

If I have two different statistics.csv files from two different servers, would it be possible to merge them somehow and generate stats for the two servers.

Trench · 2018-05-04 05:10:33

Black Mamba wrote:

If I have two different statistics.csv files from two different servers, would it be possible to merge them somehow and generate stats for the two servers.

It "looks" like it should be fine? It does appear to be "intended" for a single server's stats, but it seems like the only thing that would really be affected in a way that creates misleading results is the server stats for "time of day".

In the second .CSV file, just skip the first two lines (the "header rows" that simply describe what's in each field) and cut-n-paste the rest of the file content into the end of the first .CSV and see how it goes. Using a copy of the .CSV files and not the originals, of course.

-Trench

EDIT: A word

Last edited by Trench (2018-05-04 05:11:10)

Black Mamba · 2018-05-04 05:12:41

Trench wrote:

Black Mamba wrote:
If I have two different statistics.csv files from two different servers, would it be possible to merge them somehow and generate stats for the two servers.
It "looks" like it should be fine? It does appear to be "intended" for a single server's stats, but it seems like the only thing that would really be affected in a way that creates misleading results is the server stats for "time of day".
In the second .CSV file, just skip the first two lines (the "header rows" that simply describe what's in each field) and cut-n-paste the rest of the file content into the end of the first .CSV and see how it goes. Using a copy of the .CSV files and not the originals, of course.
-Trench
EDIT: A word

Thanks, I'll try that!

bud · 2018-05-04 07:15:59

Nice work, i had never thought of that! i opened the statistic.template for o looksie, but couldnt see any obvious problem with it that could lead to this problem, neither with the run.cmd. So i guess it might be the "bfstats.jar" thats causing it.

I found a little gem in that template tho (lol)

-- player names filter. this regular expression will be matched against the player name. if it matches the player
-- will NOT show up in the statistic. so it's kind of a "ban list" (you can even rule out your most hated clan :-) )
--
-- the example below will match against any player that has "fuck", "Fuck" (or the l33t versions like "Fcker") in
-- their name or is named "Player" or "player"
-- a good introduction on regular expressions can be found here: http://etext.lib.virginia.edu/helpsheets/regex.html
<VAR:BAN_PLAYERS>.*[Ff]u*ck.*|[Pp]layer</VAR>

Black Mamba · 2018-05-04 07:26:43

Does anyone know the formula that is used to calculate the "Score" column on the generated HTML pages?

Hydra{SRB} · 2018-05-04 13:19:46

http://bftracks.net/appl/bft/Home/view

Trench · 2018-05-04 14:23:57

Black Mamba wrote:

Does anyone know the formula that is used to calculate the "Score" column on the generated HTML pages?

I assumed it's just addition of the "Score" value logged for the player at the end of each round. i.e. Their score, versus their kills, versus their deaths. Each one of those values are being separately added together to create a total core, total kills and total deaths for all of the the rounds that have been logged for this player.

What are you observing that suggests it might be something other than that?

-Trench

EDIT: Clarification of "Are all simply being added together."

Last edited by Trench (2018-05-05 17:06:17)

SiMPLE | Forum

Announcement

#1 2018-04-25 05:41:13

Problem using BFStats to generate server stats

#2 2018-04-26 15:24:03

Re: Problem using BFStats to generate server stats

#3 2018-04-27 05:08:07

Re: Problem using BFStats to generate server stats

#4 2018-04-27 07:58:35

Re: Problem using BFStats to generate server stats

#5 2018-04-28 05:49:03

Re: Problem using BFStats to generate server stats

#6 2018-04-28 06:29:08

Re: Problem using BFStats to generate server stats

#7 2018-04-28 10:49:50

Re: Problem using BFStats to generate server stats

#8 2018-04-28 16:14:05

Re: Problem using BFStats to generate server stats

#9 2018-04-28 19:59:55

Re: Problem using BFStats to generate server stats

#10 2018-04-29 03:55:49

Re: Problem using BFStats to generate server stats

#11 2018-04-29 23:31:49

Re: Problem using BFStats to generate server stats

#12 2018-05-01 01:23:29

Re: Problem using BFStats to generate server stats

#13 2018-05-01 18:35:09

Re: Problem using BFStats to generate server stats

#14 2018-05-02 01:50:57

Re: Problem using BFStats to generate server stats

#15 2018-05-04 04:08:35

Re: Problem using BFStats to generate server stats

#16 2018-05-04 04:38:31

Re: Problem using BFStats to generate server stats

#17 2018-05-04 05:10:33

Re: Problem using BFStats to generate server stats

#18 2018-05-04 05:12:41

Re: Problem using BFStats to generate server stats

#19 2018-05-04 07:15:59

Re: Problem using BFStats to generate server stats

#20 2018-05-04 07:26:43

Re: Problem using BFStats to generate server stats

#21 2018-05-04 13:19:46

Re: Problem using BFStats to generate server stats

#22 2018-05-04 14:23:57

Re: Problem using BFStats to generate server stats

Board footer