Todkapuz Thread Counter v5

From whuckaba
Jump to: navigation, search


Todkapuz Thread Counter (08SAIMO1) is a DOS based program intended to count and do minor statistics around the SAIMOE Tourny, orignially developed in 2006, still usable in 2008.

Last v5 Beta Release: 5.98

THIS PAGE IS OBSOLETE. Todkapuz Thread Counter v6 has the current version.


Legal Stuff

Disclaimer

This program is provided “as is”. I make no warranties or guaranties, written or implied, for the use, or misuse, of this program. I do not recommend obtaining this program from any other source other than this website.

The use of this program is for entertainment purposes only.

As the program does not have access to a valid vote code record, this program will never be 100% accurate.

License

This program is provided as freeware for nonprofit and personal use, however the source code is not currently being provided as freeware. This program is not intended, nor licensed, for commercial usage. Please contact me for use within a commercial program.

Donations are welcome (contact me on AnimeSuki at Todkapuz).

Concept

Saimoe is an annual festival to share our favorite anime women with each other. It is intended to provide visibility to lesser known series, and to provide kinship for the more known series.

As part of the Saimoe festival, during the primary tourney, different anime women characters compete to determine who is the most Moe. This is done in a public forum (2ch.net) so the workings and support are all visible to all who partake.

The official scoring program uses several rules to validate votes. Primarily, the use of unique vote codes from unique posting addresses. While the valid vote codes are not released until after the polling is complete, the other aspects are available and clear to the public. This program uses those items to help remove “fake voting.”

This program is still subject to malicious fake voting, pre-voting, and admin corrections, so it will never be 100%. But it will remove most of the unwanted fake votes, and provide a reasonably close result table to use for various other purposes.

About the Various Files

08SAIMOE.EXE

  • Required to Run
  • Filename may very slightly, based on version identity.
  • File size is around 100 kB. See distro section for sizes to verify download.
  • Info: DOS based execuitable. It can be run from windows by double-clicking.

match.txt (For versions 5.5 - 5.92)

OBSOLETE

SEE HISTORY OF THIS PAGE

match60.txt (version 5.93 and up)

  • Required to Run (version 5.93 and above)
  • For Beta Version 5.93 and above
  • Is NOT back compatable to 5.92 or below.
  • Some values not compatible for 5.97a or below. See line details.
  • This file is designed to be forward compatable with anticipated 6.0+ features
  • Info: This file basically sets up the general background for the operation of the program for the match it is to parse.
  • Example:
-1
2
0
-1
-1
-1
-1
-1
0
0
01:00:00
23:00:59
0
1
in.txt
[[AS7-/Ria5eVO-HD]]
AS7-
9
1,Kokoro
1,Ranka
1,Mayu
2,Mika
2,Siesta
2,Mikan
3,Horo
3,Asuka
3,Gekka
1,<<ŒjS—School Days ƒVƒŠ[ƒY>>
2,<<ƒ‰ƒ“ƒJEƒŠ[—ƒ}ƒNƒƒXFRONTIER>>
3,<<ŒŽ‘º^—R—‚²D‚³‚Ü“ñƒm‹{‚­‚ñ>>
4,<<ˆîXŒõi‚Ý‚©‚ñj—‚ª‚­‚¦‚ñ‚ä[‚Æ‚Ò‚  ‚܂ȂуXƒgƒŒ[ƒg!>>
5,<<ƒVƒGƒXƒ^—ƒ[ƒ‚ÌŽg‚¢–‚ `‘oŒŽ‚Ì‹RŽm`>>
6,<<Š‹é‚Ý‚©‚ñ—ƒŒƒ“ƒ^ƒ‹ƒ}ƒMƒJ>>
7,<<ƒzƒ—˜T‚ƍh—¿>>
8,<<ç–ì–¾“ú‰Ä—ƒLƒ~ƒLƒX pure rouge>>
  • File format:

Line 1

This is the USE THESE SETTINGS option. If the value is '-1' then the 11 lines of configuration will be applied to the run, and there will be no real-time options. If value is '0' then next 11 lines are ignored.

Line 2

VISUALIZATION \ MODE Option. See Visulation \ Mode section for details.

Line 3

APPEND to Previous Output? See Operation Section. '-1' for yes, '0' for no.

Line 4

PRINT HEADER? See Operation Section. '-1' for yes, '0' for no.

Line 5

Allow No Double Codes? See Operation Section. '-1' for yes, '0' for no.

Line 6

Allow No Double IDs? See Operation Section. '-1' for yes, '0' for no.

Line 7

Skip No Code Entries? See Operation Section. '-1' for yes, '0' for no.

Line 8

  • Revised 5.98
  • Restrict to recording matches between times specified. '-1' for yes, '0' for no.

Line 9

RESERVED Config-9, should be '0'

Line 10

  • Revised 5.98
  • 5.98 and above: Memory / File Caching. 0 = file [stable], -1 = memory for VOTE CODES, -2 = memory for vote codes and post IDs.
  • 5.97a and below: '0'
  • WARNING: SEE MEMORY USAGE WARNING

Line 11

  • Revised 5.98
  • Version 5.98 and above: Start of Match, XX:XX:XX format. 0 = default 01:00:00
  • Version 5.97a and below: Value must be '0' or program may chrash

Line 12

  • Revised 5.98
  • 5.98 and above: End of Match, XX:XX:XX format. 0 = default 23:00:59
  • 5.97a and below: Must be '0', or program may crash.

Line 13

  • Reserved. Should be 'MASTER.TXT'

Line 14

Total Number of Input files.. should be more than 0.

Line 14a

The actual files to open for the input source (former in.txt), one each line, as many as specified in line 14.

NOTE: MUST BE IN DOS FORMAT! That means that it can only be 8 character file name, dot, 3 charector extension. If the file is in another directory, directory names must be the 8-char version.

So I highly recommend that you keep the input in the same directory as the program, and use file names like:

  • in.txt
  • in2.txt
  • in3.txt
  • 080902a.txt

Line 15

This is an example of a proper vote code. In 2007, there was a match where the vote code lengeth changed, causing issues with the program, thus the introduction of this in the match file. It should include the double brackets.

Line 16

Prefix code. This is to verify the code is one that is used for this match. Changes each day.

Line 17

Number of charactors to parse. This number must be between 1 and 9. While it is known that 2007, there were 2 matches that had 10 total chara, this is rare, and already streaches the limits of the DOS enviornment.

Lines 17a

The next (Line 17) number of lines are for the charector naming convention and match. Format:

MATCH,NAME

One Character Name per line. These are the names of the characters in the match. Do to software limitations, ROMAJI (English) is recommended. MATCH is either 1, 2, or 3. For the lines that follow this, charactor 1 is the first line, charactor 2 is the second line, etc.

All remaining lines

The following lines (up to 100 total) are match templates. in format

#,LOOKUP-VALUE

Where the number refers to the charactor it is to credit, and the item is the thing to be searched for. Note, 2ch uses Shift-JIS, and this program uses 8-bit ASCII. So in this file it will look somewhat like jibberish. I find that verifying it by copying the match file and renaming to an .htm and then opening with a web broswer capable of viewing. The use of the most recent version of Thread-Count-Light can also aid in this process. Current Alpha has experimentation of auto-discover that will help this process.

Notes

It is important that the "EOF" is present at the end of a line, not an empty line. If it is on an empty line, the program can crash. If the number in line 1 does not match the charactors actually listed, the program may crash, or produce unusable results.

in.txt (or equiv)

  • Required to run.
  • File name is set in match.txt file, so this may be whatever is selected.
  • Provided by user from 2ch.
  • File may have other names in the version 5.94+ multi-file support mode (for improved ease).
  • When viewing a COMPLETE 2ch thread, right click and 'view source'. Save source, and rename to in.txt (or similar). Basically all this is. For multiple threads, for version 5.93 and before, they must be copied into one file (end to end). Version 5.93 and before CAN detect the thread changes, and will produce a validation code similar to the actual validation file. Version 5.94 and above supports multiple input files. NOTE, however, they must include the FULL source code of each page, or it might not detect thread changes.

SAIMOE.LOG

  • Generated by program on run. Automatically over-written each run.
  • Is mostly inteded for debugging, but does include some useful information, mostly around the actual processing of the entries. It will let you know about all the fake votes it encountered, and why it was removed from consideration.

MASTER.TXT

  • Generated by program on run. User selectable on if it is appended or over-written.
  • It can be changed to .csv and opened with just about any spreadsheet program, like Excel.
  • TIME = Time of the POST
  • THD_POST = post code, similar to how the actual system works. Thread #, then 3 digit post nubmer.
  • POST_ID = Poster's id according to 2ch.
  • POST_CODE = Poster's code, hopefully as issued by the admin of the tourny.
  • Following are in vote, total vote format. Note, total votes are based on thread position, so not as useful if you are going to graph. One of my future projects is to create a stats version that will make a very nice time versus votes unit to make this quicker to graph.

CODELIST.LOG

  • Mostly a debug thing. This shows all the post codes the system encountered. Is re-written each time the program is run.

IDLIST.LOG

  • Mostly a debug thing. THis shows all the ID codes the system encoutnered. Is re-written each time the program is run.

SMRESL.TXT

  • Program Generated. New to 5.9
  • This is just the text results output.
  • Name is first
  • Total votes is next
  • Numbers in paran are the votes by each detection id
  • Numbers in brackets are fake votes tossed for that chara.

SM2WAY.TXT

  • Program Generated. New to 5.9
  • This is the results from the 2-way calculation. See Analysis Section.

SMVALI.TXT

  • Program Generated. New to 5.91
  • This is the output from the code validation routine (still in log file though).

SM3WAYM.TXT

  • Program Generated. (New to 9.56)
  • This is the 3-way match crossing, across matches only. First set is by match, second set is listed from higest to lowest.

Future Files

The 3-way crossings (both variants, hopefully) will be logged

Operation

Simply edit the match.txt file and in.txt, and run the program. Pretty Straight forward. The program then will ask a series of questions (BETA 5.9) that will determine how the program is run:

Display Choice

  • See visual display section. Use T if you don't care to see pretty stuff.

Append

  • Added Version 4.01
  • This is one of my work-arounds for mutliple files, and will probably be removed when that is resolved.
  • Appending does not allow stats to be performed, but does let you build the master.txt from several in.txt files.
  • This function is mostly historical. For all intensive purposes multiple file support negates the need for this option. Alas I leave it for the fun of BETA at the moment.

Header

  • Added Version 3 (20071007 build)
  • If you chose append, default is N, if you didn't append, default is Y.
  • Up to you if you want it for the file. I feel it is useful.
  • Header is required for Stats. Again this is mostly a historical BETA option.

Duplicate Codes

  • Major change in Version 4.5
  • Using this necessitates POST code checking.
  • This is to fight a fake vote method that was appearing where people simply copied other people's posts.
  • NOTE: Sadly the program crashed consistantly in testing when I did it in memory only (remember it only can use conventional 640k memory, or what ever it is)... so it has to disk cache... so its SLOWER... but still well worth the time investment. (Running on a fast/raid disk helps)
  • Remind me to try in-memory crunching with the new compiler. I think it is able to take advantage of "Extended Memory".

Time Control

  • Added in Version 5.97
  • Bug fixed in 5.97a
  • Manual and File Control 5.98
  • Controls the Start and end times, to keep only valid votes within the actual match period, now that multiple file usage is available, this is the next logical step.

Duplicate IDs

  • Added Version 4.08
  • This requires No Code to be on.
  • Version 4.5 - due to memory issues, had to be changed from memory based to disk based. Slowing processing time.
  • Well worth the time investment.

No Code

  • Version 4.02
  • This simply removes votes that dont have a code... first fake vote removing tool.

Running

After the questions, the program will process and analyze as applicable.

Visual Display / Mode

This is the most enjoyable part for me. Since the processing has slowed down the processing, this allows you to get a visual representation of what is happening. Time in seconds is based on a test run for Pre-release Beta 5.98 of each mode performed 09/21/2008.

No Interaction

  • 15 Seconds in File Mode, 9 Seconds in Memory Mode
  • Realtime Selection Code: Not Available in Real-Time Selection
  • File Selction Code: -2
  • Basically doesn't show anything other than where in the timeline it is in processing. No visual goodies.
  • Does NOT stop for user interaction. If you only care about getting the results in the files.

Throttled Up

  • 15 Seconds in File Mode, 9 Seconds in Memory Mode
  • Realtime Selection Code: T
  • File Selction Code: -1
  • Basically doesn't show anything other than where in the timeline it is in processing. No visual goodies.

Original Mix

  • 35 Seconds in File Mode, 20 seconds in Memory Mode
  • Realtime Selection Code: 0
  • File Selction Code: 0
  • This will show last time the chara got a vote, if it was a vote tis time, and total votes, as they are collected. This is the original visual information in pre Version 5.0

BarGraph w/o Fakes

  • 35 Seconds in file, 20 seconds in Memory Mode
  • Realtime Selection Code: 1
  • File Selction Code: 1
  • This will show all items as the Original mix, with a visual bar graph representation of where the votes are, and percentages. Very cool to see streaks late in a match that alter the playing feild. Bar graphs scaled based on highest percentage. Note, percentages are of ALL votes, not by match, which might be added in the future.

Bar Graph w/ Fakes

  • 35 Seconds in file mode, 20 seconds in memory mode
  • Realtime Selection Code: 2
  • File Selction Code: 2
  • Same as above, but also adds in the graph for fake votes in red. In a fair match, ths fake votes probably will not even be enough to show... but in some matches it is impressive.

Waterfall

  • 36 Seocnds in file mode, 22 seconds in memory mode.
  • Realtime Selection Code: 3
  • File Selction Code: 3
  • Significantly changed in version 5.9.
  • This is my favorite method. It shows a graphical representation of the votes. Shows when streaks are occuring, and where fake votes are. If you really want to get a feel fore the match, this is the way to go.

Analysis

Straight Results

This is the oldest Analysis performed by this program. Nothing terribly exciting here.

2-Way Cross, Full

The Beta 5.9 is a rework of the Beta 5.5 system (desigend for 2 girls per match, 2 matches) for the whole 9 girls. Note, this is still 2-way picking. So if you sum a row and column, it will not match the total votes for that girl (should be higher). This is because if someone picked girl set 1-4-7 ... that is seen to the program as 1-4, 1-7, and 4-7 ... all 3 of the 2-way crosses. Thus, if summed, you'd have 2 1s, 2 4s, 2 7s. Numbers in brakets are votes cast ONLY for a given character, with no cross (ie, they didnt vote in any other match). 2-way cross is still primarily intended for 2 matches at a time, not 3, but still can provide interesting popularity information. This data should only be used for that type of analysis... ex: "People who liked 1, also seemed to vote for 7." etc.

There are 36 total possible crosses + 9 singles for a total of 45 data points.

Post Numbers

For Beta 5.9 this is located in the Saimoe.log file. This is the same type of result the actual processing unit uses to show what votes counted. This is intended to serve as both a way to see if it counted your vote, as well as a chance to compare against the ACTUAL results once released, to look for votes that were missed or counted by the program that were or were not real.

2-way Cross, Match (Not Implemented)

There really isn't a big reduction in data for doing match based. I probably will not pursue this at this time.

3-Way Cross, Full (Not Implemented)

This is currently under development. The difficulty is there are approximately 252 possible 3-way crosses (full), + 36 2-ways + 9 1-ways that total 297 data points. This is not available in Beta 5.9

3-Way Cross, Match

Available in 5.98. The sysetm will allow 4 x 4 x 4 match runs (64 options). But note, it will throw away anything that is NOT a 3-way match, or where two from the same match are chosen.

Results are available on-screen (non-pass-through) and by output file.

Change Log

5.98 Beta (from 5.97 Beta)

  • Program Body Reduced from 739 lines to 703 lines (with the addition of around 20 new lines to body).
  • Total Program now 2439 lines, 19 sub-routines, 13 variable arrays, countless variables and strings.
  • Controlability of Time ... 5.98 allows end user to control when it will start and end the recoding of votes.. 01:00:00 to 23:00:59 are default if you error.
  • USAGE OF MEMORY IS BACK. TTC used memory to hold vote codes many many versions ago, but as the number of variable arrays and strings increased, the program ran out of "standard" memory, and would crash. However, with the new compilier, it appears that the program can take advantage of more extended memory. Old Compiled version crashes after ~300 counted votes ... I have not yet foudn the crash point for new complier... but it can handle my 703 test routine. NOTE: I CAN NOT PROMISE STABILITY WITH MEMORY USAGE beyond 703 votes.
  • Over-all graphics handling has been refined, with margin efficieny increase
  • Some redundent coding has been pruned, and all validation routines have been moved to sub routines.
  • No known way to work around Window's craziness with drag-drop within the program, so have to use link if you wish to use this functionality.
  • Fixed double water-faller hit on fake votes... marginal processing improvement.

5.97a Beta (from 5.97 Beta)

  • In the words of Apple: "Bug Fix"
  • Not necessary to download. No improved functionality... just a few minor glitches in certain circumstances.

5.97 Beta (from 5.96 Beta)

  • Addd drag-n-drop capability for Match60.txt file... NOTE, windows gets weird with this, so to use this, FIRST you must create a shortcut to the EXE file (left drag, create short-cut)... this will keep the directory local, otherwise it uses the documents and settings folder, which messes everything up.. then drag your match60.txt (or equiv) file onto the short-cut.. this will allow you to use files other than match60.txt if you want to have multiple configurations all ready at one time.
  • Added HARD CODED time controller... I will need to work core of the program to help streamline these checks, as the actual middle of the program is becoming quite bulky, beyond understanding. I will be adding a new variable set, to allow times over then 01:00:00 to 23:00:59, if so desired.

5.96 Beta (from 5.94 Beta)

  • Added 3-Way Simplified Statistics Set.
  • Did some pruning of the Saimoe.LOG output... although 3-way now is quite noisy (in test data anyhow).

5.94 Beta (from 5.93 beta)

  • Now natively supports multiple input files.

5.93 Beta (from 5.92 beta)

  • Adoption of MATCH60.TXT, and obsolete of match.txt.
  • Validation Results no longer in SAIMOE.LOG
  • Two-way Stats no longer in SAIMOE.LOG
  • Bar-graph visulations now scale to maximum percentage, allowing to see fine detail (instead of fixed 50%)
  • Addition of 'No Interaction' feature. This allows the program to be called and run without any interaction on the user's part once running.
  • Force Setting Mode... you now can save your favorite configuration set in the MATCH60.TXT file, and use it, if you find that easier.

5.92 Beta (from 5.91 beta)

  • Continuity Bug fix.
  • Configuration match.exe moved to subroutine in preperation for file-config.

5.91 Beta (from 5.9 Beta)

  • Compiled on Compiler version 7.1 (as opposed to 4.5) - fixes string memory problem in compiling (or appears to for now anyhow)
  • SMVALI.TXT is now created
  • BugFix - Header and Framework in 2-way compairison
  • BugFix - Linedown after results showing
  • Code Validation visual improvement
  • Moved Config to subroutine to prepare for 6.1 implementation of file-based config
  • Moved validation to subroutine
  • Primary program reduced from 1036 to 743 lines
  • 8 seperate config strings combined into 1 array.

5.9 Beta (from 5.5 Beta)

  • COSMETIC - Rebranded 08Saimoe
  • COSMETIC / MATH - Water fall graphic changed. Original color gradiation was dark blue, medium blue, bright blue, white, done ad-hoc where as the votes scrolled off the screen, the trends actually went down... which probably wasnt visible at typical operating speed. New color gradiation calculation performed at adding to first row, not ad-hoc .. this will add more time to the calculation, but then again the water-fall is not intended to be "fast". The way this new calculation works, fake votes will NOT interrupt trends, and there can be 2 open spots (so pingpong doesnt effect rally), however this negatively affects rally colorations (which I think is a good thing)... hard to explain.
Start of Ralley = Green Solid
2nd volly = dark blue 25% radient on black
3rd volly = dark blue 50% radient on black
4th volly = dark blue 75% radient on black
5th volly = dark blue 100% 
6th volly = light blue 25% radient on dark blue
7th volly = light blue 50% radient on dark blue
8th volly = light blue 75% radient on dark blue
9th volly = light blue 100% 
10th volly = cyan
11th volly = WHITE
  • Files - Added SMResl.txt - Results Only File, Added SM2way.TXT - 2way permutation
  • USABILITY - command line /E command hasn't worked for awhile... deleted the statement. However the other commandline cheat has been re-enabled to by-pass questions. Can you guess the secret?
  • STATS \ Programing - 2-way increased to include all 9 chara. The 2-way stats has been matrixed to provide all possible 2-way combos. Note, due to the way the alg is setup, it WILL report more votes over-all than actually occured where 3-chara get votes at a time... as it will see A-B-C as though A-B was 1 vote, and A-C was 1 vote, and B-C was 1 vote. So don't think of it as votes, persay, but times someone chose a particular combo. (I hope that makes sense). This and another sub routine were completely re-written to allow the program to be compiled. The mainbody of the program will still need to be worked to allow for 3x3.

In the Works

  • Change the Results to work with Match data, now that such data is available
  • Rework detection routine to allow for upgrade below. (This will be, hopefuly, unnoticable to user)
  • Do "best guess" matching, to ease work on front-end coding by user
  • Incorporate TCL into this program to aid in making the match60.txt file
  • let user decide what result talleys to perform (0 = all).
  • do 2way that sorts out highest to lowest.. just to be nice.
  • match60.txt is going to have to change.... either combining the validation part or something, so that I can do adaptive and do selective results.

Bugs

  • Drag and Drop usage forces "?:\documents and settings\USER\" to be the default start location... so MUST use shortcut to get this to operate correctly.

What you Really Want

IMPORTANT - RIGHT CLICK, SAVE-AS ... otherwise the .txt will just load in this window. Save to any directory (preferably one just for this program).

CURRENT

  • In.txt - Example - UPDATED FOR 5.98 no longer available due to v6
  • [match60.txt] - EXAMPLE - UPDATED FOR 5.98

HISTORICAL