04 00 00 00 is for blank frames clearly judging by WW shps and the ww tool test and not surprised it glitches when give content.
fog.shp/shroud.shp can't be compressed anyway, its an alpha process and those do not have compressed blitter variant so misdraws if was compressed.
Not sure why you thought its compressed TBH.
BTW, the stripe bug is AFAIK unfixable in the shp itself as game itself has a hard coded limit stupidly set in place (ARES fixes this WW programmer oversight) for the compressed line and past that reads stuff normally than process compressed as usual.
00 is same as 01, no difference ingame.
02, there is not single shp in the game I have ever seen using it and would gladly wanna see any official shp that has it thus likely invalid.
Frankly as interesting this 'poking randomly' is, there is not likely any improved compressions I'd be interested in achieved here as RLE compression method by WW is inefficient sadly as even simple PCX's RLE does better if give it same cropped source!
It is pity they did not bring LCW to TS for good compress TBH or at very least RLE the content as well and forget stupid line lenghts declaring which only wastes space. QUICK_EDIT
Implemented a "Ignore Boring Frames" edition of the debug. Now with less boring data.
Basically frames (And files) which are the bog standard are ignored. So any frame which either has no Width/Height or which has compression as 1,2,3 and zeros for the other 3 bytes. Only the interesting ones are left.
Implemented Shadow Support. Supports shadows on SHP's that are classed as Unit, Building or Building Animation. Shadows can either be rendered in their normal colour or with partial transparency (And black colour). Shadows will always be shown when you hit the animate button or when your using the Open File Dialog (However the bit that lists shps can only do RA2 units since it only loads a single frame).
Implemented the remaining TD/RA1 decode code (Aka nicked the Decode40Tri from SHP Builder). Currently has no ability to save TD/RA1 SHP's.
Also updated the Side Colour code to apply the colour to the correct RA1 palette location.
P.S I added code to check if a Unit is a RA2 unit by scanning the first frame for shadow colours (Since TS never has shadows in the first frame, nor the same shadow colour). So it should always work out TS/RA2 units correctly (At least official ones). Also added code to fix buildings getting the Animation Type by checking for shadows. _________________ Free Map Editor - Game Requirements - Stucuk.Net QUICK_EDIT
Have you ensured that other tools correctly read your files as well? You idea to add a tailer that contains palette and other information is probably a bad idea regards compatibility with other tools, I know that XCC validates the file size for TD/RA shape files at least (though you aren't writing those yet).
You should also probably look at the improvements I made to the TD/RA Shape encoder in XCC for implementing that format. QUICK_EDIT
P.S I added code to check if a Unit is a RA2 unit by scanning the first frame for shadow colours (Since TS never has shadows in the first frame, nor the same shadow colour). So it should always work out TS/RA2 units correctly (At least official ones). Also added code to fix buildings getting the Animation Type by checking for shadows.
IIRC only RA2 infantry used shadow on the first frame, but then not index #1 but index #12 (pure black) color.
TS and RA2 use both palette index #1 for shadow (#0 being transparent). Since the real color plays no role and depends on the loaded palette, i don't see how you want to determine the shadow color only by looking at the SHP, nor how this allows you to find out for which game the SHP was made.
You can use an SHP made for TS and show it exactly the same in RA2, if RA2 has the same palette. So from an SHP point there is absolutely no difference between the games. _________________ SHP Artist of Twisted Insurrection: Nod buildings
Note: I haven't built the debug build. So ignore it. Also may have some memory leaks as i haven't built the debug build to check.
Blade wrote:
Have you ensured that other tools correctly read your files as well? You idea to add a tailer that contains palette and other information is probably a bad idea regards compatibility with other tools, I know that XCC validates the file size for TD/RA shape files at least (though you aren't writing those yet).
As long as the games work fine with it i'm happy. I won't be dictated to by 3rd party tools which were designed badly(If people "Obey" the file format there should be no issues). However i do plan on making the extra data being saved optional when i do a custom save dialog (There is already a FSaveExtra in the TSHP class which determines if it saves the extra data).
Lin Kuei Ominae wrote:
IIRC only RA2 infantry used shadow on the first frame, but then not index #1 but index #12 (pure black) color.
TS and RA2 use both palette index #1 for shadow (#0 being transparent). Since the real color plays no role and depends on the loaded palette, i don't see how you want to determine the shadow color only by looking at the SHP, nor how this allows you to find out for which game the SHP was made. Confused
Code:
if (FType = stUnit) and (FGame = sgTS) then
if ScanFrameForColor(12,0) then
FGame := sgRA2;
if (FType = stAnimation) then
if ScanFrameForColor(1,(High(FFrames)+1) div 2) then
FType := stBuildAnim;
Just checks for Colour #12 on the first frame if its classed as a unit. Currently only treats #12 as the shadow colour if its a RA2 SHP and has the Unit Type. Everything else with TS/RA2 it uses colour #1. With RA1 it uses #4. I havn't extracted TD files yet so it currently ignores TD.
Lin Kuei Ominae wrote:
You can use an SHP made for TS and show it exactly the same in RA2, if RA2 has the same palette. So from an SHP point there is absolutely no difference between the games.
Which is why it tries to determine the Game as well as Type. You then have a higher chance of automatically picking the correct palette.
When animating the Type comes into play as Units/Buildings/BuildAnims all have shadow frames to skip when animating (As in consider the end of the animation where the shadow frames start). _________________ Free Map Editor - Game Requirements - Stucuk.Net QUICK_EDIT
from where do you get FType and FGame? There are no such variables in the SHP afaik, except you mean already your added extra data from the end of the file.
That whole "guessing" already made SHP Builder quite a pain to work with, until Banshee added the option to remember the last used palette and thus allowed to disable the not working SHP type guessing.
It's simply that the guessing doesn't and can't work well enough.
Especially not with all the custom made SHPs.
And when you are later forced to constantly select your working palette, because the program guessed a wrong one again, this can become quite annoying.
When you are working with a tool, it's a lot more likely that you keep using the same chosen palette, than constantly a different one.
Not to mention that nowadays we use 90% custom palettes. So the whole guessing can't work at all anymore. _________________ SHP Artist of Twisted Insurrection: Nod buildings
BTW, the stripe bug is AFAIK unfixable in the shp itself as game itself has a hard coded limit stupidly set in place (ARES fixes this WW programmer oversight) for the compressed line and past that reads stuff normally than process compressed as usual.
No clue what Ares fixed, but changing the Flags actually affects that, which is why i started to think its controlling Blitters in some way in the first place. I managed to break it so much the units behind the building were rendering all on top.
I'm starting to think Buildingz comes to play somewhere...
Well i do have reasonable cause to think it. ShapeSet uses the same Blitting functions as the game, this is in one of them
ApolloTD wrote:
04 00 00 00 is for blank frames clearly judging by WW shps and the ww tool test and not surprised it glitches when give content.
fog.shp/shroud.shp can't be compressed anyway, its an alpha process and those do not have compressed blitter variant so misdraws if was compressed.
Not sure why you thought its compressed TBH.
BTW, the stripe bug is AFAIK unfixable in the shp itself as game itself has a hard coded limit stupidly set in place (ARES fixes this WW programmer oversight) for the compressed line and past that reads stuff normally than process compressed as usual.
00 is same as 01, no difference ingame.
02, there is not single shp in the game I have ever seen using it and would gladly wanna see any official shp that has it thus likely invalid.
Frankly as interesting this 'poking randomly' is, there is not likely any improved compressions I'd be interested in achieved here as RLE compression method by WW is inefficient sadly as even simple PCX's RLE does better if give it same cropped source!
It is pity they did not bring LCW to TS for good compress TBH or at very least RLE the content as well and forget stupid line lenghts declaring which only wastes space.
The point of that image was to show that non-compressed shps can be viewed as raw data, and the point of that is was to offer a different perspective on the files removing the factor of the over a decade old baseless assumptions that compression is specified in the shp header.
There isn't, there's just RLE, it's possible there might be variations of RLE not just RLE Zero, but from what i saw when Bittah tested the flags i doubt that.
WW made in the way they needed it. The RLE just compresses Zeros, as far as its efficiency goes it does it properly.
ShapeSet doesn't show any proof of any other compressions either.
Hyper looked into that hidden -X command that set flags to 1 and added extra two bytes with 00's in my test shp, it would seem it's related to shp's created from DPaint Animation files.
Yea as far as LCW goes its in the exes but its only used for VQA's and some legacy code that the game itself no longer uses. _________________ Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1
Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite Last edited by tomsons26lv on Mon Aug 22, 2016 11:28 pm; edited 2 times in total QUICK_EDIT
Have you ensured that other tools correctly read your files as well? You idea to add a tailer that contains palette and other information is probably a bad idea regards compatibility with other tools, I know that XCC validates the file size for TD/RA shape files at least (though you aren't writing those yet).
As long as the games work fine with it i'm happy. I won't be dictated to by 3rd party tools which were designed badly(If people "Obey" the file format there should be no issues). However i do plan on making the extra data being saved optional when i do a custom save dialog (There is already a FSaveExtra in the TSHP class which determines if it saves the extra data).
Much as some of the code in XCC isn't easy to follow, you can hardly claim its badly designed for not "Obeying" the file format when you yourself are blatantly planning on creating out of spec files. The general rule when creating tools to work with a file format is be a flexible as possible when reading them while being as strict as possible when writing them.
There is also the issue that this wastes memory in game as shape files are loaded into memory in their entirety. Not that its a massive issue with he GB's we have these days, but I would definitely say people shouldn't distribute shape files with the extra information in them for these reasons.
At the end of the day its your project and you can take it in whatever directions you want, but you already complained hardly anyone contributed to Shp Builder but being dismissive of other people's opinions and feedback and proclaiming that you won't be beholden to 3rd party utilities handling of the files doesn't sound to me that you are too concerned about getting other people on board with supporting it. QUICK_EDIT
from where do you get FType and FGame? There are no such variables in the SHP afaik, except you mean already your added extra data from the end of the file.
Its not anywhere in the format... its part of the TSHP class.... Source code is included.
Lin Kuei Ominae wrote:
That whole "guessing" already made SHP Builder quite a pain to work with, until Banshee added the option to remember the last used palette and thus allowed to disable the not working SHP type guessing.
It's simply that the guessing doesn't and can't work well enough.
Especially not with all the custom made SHPs.
And when you are later forced to constantly select your working palette, because the program guessed a wrong one again, this can become quite annoying.
When you are working with a tool, it's a lot more likely that you keep using the same chosen palette, than constantly a different one.
Thats the whole point of the Extra data added to the SHP file. It saves all of it...
Lin Kuei Ominae wrote:
Not to mention that nowadays we use 90% custom palettes. So the whole guessing can't work at all anymore.
Saved as extra data....
Blade wrote:
Much as some of the code in XCC isn't easy to follow, you can hardly claim its badly designed for not "Obeying" the file format when you yourself are blatantly planning on creating out of spec files. The general rule when creating tools to work with a file format is be a flexible as possible when reading them while being as strict as possible when writing them.
Given how SHP files are layed out, there is no reason to read beyond the frame data. Any application which randomly tries to load unknown data (Or randomly assumes it has to be a certain bit of data) which isn't in the specification is badly coded. You should always follow the rules of a file format and never guess. You can take most file formats and add data at the end of them without issue (EXE's for example). XCC 1.47 and SHP Builder (Any version) doesn't seem to have any issues loading the SHP's (There is no reason for any tool to have an issue). Where is the phantom SHP tools which crash when they load a SHP with extra data?
Unless you find a tool which has an issue with extra data at the end of a shp then its pointless arguing about it.
Blade wrote:
There is also the issue that this wastes memory in game as shape files are loaded into memory in their entirety. Not that its a massive issue with he GB's we have these days, but I would definitely say people shouldn't distribute shape files with the extra information in them for these reasons.
778 Bytes extra data, equivalent to a frame with 27x27 pixels (Including the frame header). Your not seriously going to say that 778 Bytes extra per SHP is going to make a computer grind to a halt....
If you have 3,606 SHP files (Thats the total i have in my SHP folder containing SHP's from RA1, TS, RA2) you would have to load an extra 2.67MB into memory. 2.67MB is nothing even back in 2000. Your argument over increased size is completely irrelevant.
Blade wrote:
but you already complained hardly anyone contributed to Shp Builder but being dismissive of other people's opinions and feedback
What feedback have i been dismissive of? I take all feedback into account but i am not necessarily going to do everything that everyone wants.
If your referring to the "It may break stuff" then its not going to hold me back unless you find an actual case where its an issue. Everything is speculation and as it stands SHPToolz will not be used by ANYONE for anything as currently it can't do much other than view SHP's. The end of the world hasn't happened yet.
Blade wrote:
and proclaiming that you won't be beholden to 3rd party utilities handling of the files doesn't sound to me that you are too concerned about getting other people on board with supporting it.
People supporting the tool has nothing to do with another tools ability to load SHP's. As it stands there is no reason for any application that loads SHP files to do anything with any extra data found in a SHP file. Your making a big deal out of nothing.
Lets say that Tool "A" writes a SHP using format 999 which the game supports and has superior compression, but Tool "B" doesn't like format 999. Should Tool "A" drop support for format 999 simply because Tool "B" wasn't designed to handle it? Extra Data in this case should be ignored as there is no reason for a tool to load anything extra than what it expects. Something like Extra Data should be something other tools want to support so that SHP's can be rendered more properly across the board.
The main point of me writing the code is to build the TSHP class. It gives everyone the ability to load and save SHP files however they like and is full of options like not saving the Extra data to a SHP file. You can easily use it to create a tool which doesn't save extra data.
P.S SHPToolz can't actually read the Extra SHP data in the build i released as the code is wrong, it just skips it. _________________ Free Map Editor - Game Requirements - Stucuk.Net QUICK_EDIT
Also Known As: banshee_revora (Steam) Joined: 15 Aug 2002 Location: Brazil
Posted: Mon Aug 22, 2016 11:15 pm Post subject:
Lin Kuei Ominae wrote:
Not to mention that nowadays we use 90% custom palettes. So the whole guessing can't work at all anymore.
Just to point out: in OS SHP Builder, you can customize the palettes that are loaded for units, infantry, animations, cameo, etc... for each game. QUICK_EDIT
Fence Post's use it. Don't see why since 2 is the same as 1 except you have 2 bytes extra per row to tell it the width you already know.
[/url]
Okay, I checked the relevant files, it basicly means, compression 2 is ONLY used when there are none of background pixels (0) within the cropped canvas but the benefit of this compression is pointless when you just tell linelenghts all for nothing wasting space :/
Anyways the extra trailer data will get read by OpenRA sadly as pixels, already had this problem with alphatst.shp way back when outside of canvas were extra junk bytes. Obviously TS/RA2 will have no issue as reads strictly to the demands of canvas size set as far drawing goes but it will probably store whole file in RAM.
tomsons26lv wrote:
The point of that image was to show that non-compressed shps can be viewed as raw data, and the point of that is was to offer a different perspective on the files removing the factor of the over a decade old baseless assumptions that compression is specified in the shp header.
There isn't, there's just RLE, it's possible there might be variations of RLE not just RLE Zero, but from what i saw when Bittah tested the flags i doubt that.
WW made in the way they needed it. The RLE just compresses Zeros, as far as its efficiency goes it does it properly.
ShapeSet doesn't show any proof of any other compressions either.
Hyper looked into that hidden -X command that set flags to 1 and added extra two bytes with 00's in my test shp, it would seem it's related to shp's created from DPaint Animation files.
Yea as far as LCW goes its in the exes but its only used for VQA's and some legacy code that the game itself no longer uses.
Yeah sure can view uncompressed that way but frankly I can tell even with just hex editor when they are compressed and not easily. There is no variations on the RLE and no, this RLE is not efficient in terms of filespace usage. You waste 2 bytes every time you tell there is a background pixel, then you waste 2 bytes to tell the line lenght per every line. Where as PCX RLE is encoded on the principle that every byte is assumed to be single byte telling RLE count, exceptional range value to this can be used to signal when to process normally and how many normal bytes follow before switch back to normal RLE parsing.
Thus to skip 50 background pixels, PCX basicly puts value 50 while shp puts 0 and 50. INEFFICIENT. Then lets assume width of frame was 50 pixels, shp also tells it in 2 bytes making it up to +3 waste bytes compared to pcx method Outcome of this I seen is, shp (C3) ends up 2.10mb and pcx of same content 1.60mb...
That is what I mean inefficient, it sure works for the game but it is wasteful byte programming if really wanted to compress GOOD.
Yep LCW is in the exe but only for VQA actively, could have kept for SHPs but discarded since RA1 shps, likely performance issues with larger filesizes which was no issue for video files when only one gets called at a time instead of dozen. Last edited by ApolloTD on Tue Aug 23, 2016 7:46 am; edited 1 time in total QUICK_EDIT
SHP like RA2 ctbunk02.shp uses 2 palettes, unittem for image and isotem for rubble frame.
Your current extra data wouldn't account for that. Would be nice if such SHP are viewable
correctly. RA2 ctcity04.shp uses citytem palette which cannot be viewed by your tool right
now as it is not included.
Custom palette support in dropdowns is like a need for mods. Suggestion is to scan the folder
for palettes and the usage of palette names as they are like unittem instead of using fixed
names like Building or Unit. QUICK_EDIT
Anyways the extra data will get read by OpenRA sadly as pixels, already had this problem with alphatst.shp way back when outside of canvas were extra junk bytes.
Looking at the Source code i don't see that. Maybe i am looking at the wrong code but according to the following code they load SHP's "Properly"(As in sticking to the specification).
That is what I mean inefficient, it sure works for the game but it is wasteful byte programming if really wanted to compress GOOD.
I would imagine they were going for a trade off between compression and rendering speed. Rendering a simple RLE like SHP's have will be faster than a proper one.
E1 Elite wrote:
SHP like RA2 ctbunk02.shp uses 2 palettes, unittem for image and isotem for rubble frame.
Your current extra data wouldn't account for that.
It would if a new type was added specifically for it. Selecting that type would make it use two palettes. Though i'm guessing that the rubble frame is at the same index for each building? Otherwise it would be impossible.
E1 Elite wrote:
RA2 ctcity04.shp uses citytem palette which cannot be viewed by your tool right
now as it is not included.
At this stage i haven't gone through the mix's for all the palettes. Though citytem.pal should be in the latest download (Its just never used by the app).
E1 Elite wrote:
usage of palette names as they are like unittem instead of using fixed
names like Building or Unit. It would also remove any doubt whether I am using the correct
palette.
Well the Type part is not entirely just for the palette(Its only tied to the palette in relation to automatically working out the default palette for that SHP before the user selects a custom one), its also for example to work out how to animate it as buildings/units have the separate shadow frames. Everything is a work in progress however.
There will be the ability to select a custom palette (Its just not coded yet) and it will display the palette name. _________________ Free Map Editor - Game Requirements - Stucuk.Net QUICK_EDIT
Usually rubble frame is 4th in the index, but if you can mark it as the last normal frame (not
shadow) it would be better.
Problem with the fixed palette name is that it lacks theater info which creates a doubt whether
we are using the correct palette. Moreover multiple of those crowding the dropdown like Unit,
Building, Building Anim etc. refer to the same palette and we are used to seeing unittem.pal
in there. QUICK_EDIT
ModEnc seems to agree that there is a flag in the header that indicates either compression (Zero RLE) or no compression and that is bit 2 of the flags (basically and & 0x02 check).
stucuk, you complain that other tools shouldn't assume the end of the image data is the end of the file, but as it exists, that IS the spec, they aren't out of spec by doing that. Infact in the TD format header, the last image offset is to the end of the file, so its perfectly within spec to check that the flile end and that offset agree. Its you who are proposing to arbitrarily and unilaterally extend the spec and expect all other tools to play nicely with it.
Basically this boils down to the SHP format not supporting the meta data you want to associate with it and you are exploiting the fact that the game doesn't do any strict conformance checking in the same way the mix "protectors" corrupt the mix format.
A more compatible solution would be to generate a meta data file named the same as the shp file that can be distributed with it IMO (ed SPRITE.SHP gets SPRITE.TLZ or some other extension generated to go with it). That way the actual files are kept clean, your tool can have its own specialised meta data to work with and other tools will always ignore it as you intend them to. QUICK_EDIT
ModEnc seems to agree that there is a flag in the header that indicates either compression (Zero RLE) or no compression and that is bit 2 of the flags (basically and & 0x02 check).
What does 0x01 mean then? Cos its used with both no compression and Zero RLE but isn't set when its "Decode 2".
Blade wrote:
stucuk, you complain that other tools shouldn't assume the end of the image data is the end of the file, but as it exists, that IS the spec, they aren't out of spec by doing that.
The specification is the layout of the file based on what should exist given the header's data. There is nothing to state that the EOF is just after the last frame and nothing to state it isn't.
Answer me this one question. Which applications fail to load the extra data? Until you answer that question its only an issue of your making.
Blade wrote:
Basically this boils down to the SHP format not supporting the meta data you want to associate with it and you are exploiting the fact that the game doesn't do any strict conformance checking in the same way the mix "protectors" corrupt the mix format.
SHP Format doesn't specify anything after it. Thats the point. You only read what is specified and there is no reason to care if extra data is after it as you have already loaded everything the headers specified. Why can't you understand that simple fact.
If your writing good code you should never assume your reading a single file. For all your code knows it could be passed a stream that contains multiple files.
Corrupting a header is completely different to adding data AFTER the regular file. The former is designed to harm and the latter should be ignored by everything that doesn't know what it is (Since nothing in the Spec states the "BodySize" unlike the Mix Spec. Apples and Oranges).
P.S I haven't checked XCC's source, but it should just be making sure that BodySize is not larger than the amount of data in the file (Smaller should be no issue).
Blade wrote:
Infact in the TD format header, the last image offset is to the end of the file, so its perfectly within spec to check that the flile end and that offset agree.
The last offset is not to the end of the file but to the last image data (at least based on official TD SHP files). There is no reason to read beyond it or check that after reading the last frame that its the EOF. You already loaded the entire SHP file according to what the Headers told you.
If the game doesn't do a random check then why should you? Its got nothing to do with the specification to check where the EOF is as it doesn't specify where it is.
P.S ccfiles4.txt is an example of what Specifications are. They tell you what to load and how to load it. It states nothing on where the EOF is in relation to the last frame. As long as people follow the specification and don't add their own random stuff like seeing where the EOF is then there will never be any issues.
Lin Kuei Ominae wrote:
I would support that.
So would i if it was an issue. As it stands its not. I have still to hear about an application which can't handle it. So far its just Blade speculating that some random application may not be able to handle it. _________________ Free Map Editor - Game Requirements - Stucuk.Net QUICK_EDIT
So would i if it was an issue. As it stands its not. I have still to hear about an application which can't handle it.
Doesn't matter for me. I would still prefer a clean SHP with an extra data additional file. _________________ SHP Artist of Twisted Insurrection: Nod buildings
ModEnc seems to agree that there is a flag in the header that indicates either compression (Zero RLE) or no compression and that is bit 2 of the flags (basically and & 0x02 check).
What does 0x01 mean then? Cos its used with both no compression and Zero RLE but isn't set when its "Decode 2".
I have more answers on that now actually,the first byte?(could as well be a word or even dword bitfield as WW did already use those in RA) a bitfield, see the txt i keep posting but seemingly you keep missing.
stucuk wrote:
Answer me this one question. Which applications fail to load the extra data? Until you answer that question its only an issue of your making.
I tried adding extra data after the shp, as it stands i only tested it for a moment, in RA and RA2 it works, XCC says the shps are unknown, OpenRA crashes on game startup. _________________ Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1
Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite QUICK_EDIT
It is quite possible that there is no code in the game for Flag type 1, with Flag 0 being No compression and Flag 2 being Zero RLE.
The game only checks the header flag 'just' before blitting and it only checks for Flag 2, otherwise, No compression is assumed (Being 0 or 1).
I would have to investigate Shapeset more to find out how it claculates the Flags when writing a shape file.
EDIT: Due to tomsons testing, 0 is definatly No compression, 1 is ignored in TS and it seems RA2/YR has added code to handle Flag 1. No update as to what it is actually for yet. QUICK_EDIT
Infact in the TD format header, the last image offset is to the end of the file, so its perfectly within spec to check that the flile end and that offset agree.
The last offset is not to the end of the file but to the last image data (at least based on official TD SHP files). There is no reason to read beyond it or check that after reading the last frame that its the EOF. You already loaded the entire SHP file according to what the Headers told you.
If the game doesn't do a random check then why should you? Its got nothing to do with the specification to check where the EOF is as it doesn't specify where it is.
P.S ccfiles4.txt is an example of what Specifications are. They tell you what to load and how to load it. It states nothing on where the EOF is in relation to the last frame. As long as people follow the specification and don't add their own random stuff like seeing where the EOF is then there will never be any issues.
That specification is incomplete, all the official shape files have frames + 2 entries in the offset array. If there is a "loop frame" then the second to last points to the start of it and the last entry points to the end of the file. If no loop frame is present (most shape files), then the second to last points to the end of the file and the last is 0's. I've done a lot of work on the TD shape format, I assure you this spec is correct.
XCC checks this to determine if a file is a TD Shape file BTW.
I don't get why you are so invested in attaching the data to the end of the file when to any other application is just going to be extraneous junk at best, why not just keep it separate? QUICK_EDIT
Also Known As: banshee_revora (Steam) Joined: 15 Aug 2002 Location: Brazil
Posted: Tue Aug 23, 2016 9:47 pm Post subject:
Stucuk, you should stop being stubborn for a while and listen to Blade's advise. Add aditional data to an external file as he says. Do not expect people in the future may not necessarily follow your standards, even if they happen to know and use your program. QUICK_EDIT
I tried adding extra data after the shp, as it stands i only tested it for a moment, in RA and RA2 it works, XCC says the shps are unknown, OpenRA crashes on game startup.
Are you adding it to TD/RA1 SHP's or TS/RA2 SHP's? XCC states the type as "shp (TS)" when extra data is saved to a TS/RA2 SHP. Based on what blade has said if its a TD SHP it will fail if you only add extra data (Though altering the eof offset may make it work. I still need to get round to coding the TD SHP writer).
CCHyper wrote:
It is quite possible that there is no code in the game for Flag type 1, with Flag 0 being No compression and Flag 2 being Zero RLE
Ah, that makes so much sense. "Decode 2" Is "Decode 3" but it just has no zeros in it. Though i don't get why their encoder would choose to compress it when uncompressed would be smaller. There are SHP's which have both compressed and uncompressed frames in the same SHP.
@Banshee: SHP Builder doesn't treat it like a flag.
Blade wrote:
That specification is incomplete, all the official shape files have frames + 2 entries in the offset array. If there is a "loop frame" then the second to last points to the start of it and the last entry points to the end of the file. If no loop frame is present (most shape files), then the second to last points to the end of the file and the last is 0's. I've done a lot of work on the TD shape format, I assure you this spec is correct.
Then what would happen if you modify the eof offset to be the true eof. As in after the extra data rather than last frame. Also it may be possible to store data after the offsets and before the first frame starts (Depending on if it looks at the first frames offset and seeks to it or if it just starts where it left off. If the former then it shouldn't be an issue).
The question i have is if the first extra offset is where it should go after reaching the end of the animation then whats the second extra offset for?
@Banshee: SHP Builder ignores the 2 extra offsets on load and saves blank ones. If any SHP relies on the "Loop" ability then SHP Builder would invalidate it when it saves.
Blade wrote:
I don't get why you are so invested in attaching the data to the end of the file when to any other application is just going to be extraneous junk at best, why not just keep it separate?
Keeping it separate is less convenient and "floods" a directory with additional files. I prefer clean solutions where possible.
Banshee wrote:
Stucuk, you should stop being stubborn for a while and listen to Blade's advise
I will always be stu born.
Banshee wrote:
Do not expect people in the future may not necessarily follow your standards, even if they happen to know and use your program.
I never do. Its also why i have stated from the beginning that everything will be --> optional <--. _________________ Free Map Editor - Game Requirements - Stucuk.Net QUICK_EDIT
@Stucuk:
I think it is time to abandon the old names people have used in various projects. Through mine and Blades time working on reverse engineering these games and talking to ex-WW programmers, we pretty much know the names for each type of compression/codec/algo etc used in the games.
I have been spending most of my evening going through the Shapeset tool and we have a good idea where and how it writes shape files, even the RLE code it uses (we can provide C pesudo for you if you want to impliment a true compressor or decompressor QUICK_EDIT
I tried adding extra data after the shp, as it stands i only tested it for a moment, in RA and RA2 it works, XCC says the shps are unknown, OpenRA crashes on game startup.
Are you adding it to TD/RA1 SHP's or TS/RA2 SHP's? XCC states the type as "shp (TS)" when extra data is saved to a TS/RA2 SHP. Based on what blade has said if its a TD SHP it will fail if you only add extra data (Though altering the eof offset may make it work. I still need to get round to coding the TD SHP writer)..
Thats why i mentioned RA and RA2, i tried both the formats and both the games as well as ORA _________________ Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1
Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite QUICK_EDIT
Also Known As: banshee_revora (Steam) Joined: 15 Aug 2002 Location: Brazil
Posted: Wed Aug 24, 2016 1:57 am Post subject:
stucuk wrote:
@Banshee: SHP Builder doesn't treat it like a flag.
But in this case, it doesn't make any difference at all.
stucuk wrote:
@Banshee: SHP Builder ignores the 2 extra offsets on load and saves blank ones. If any SHP relies on the "Loop" ability then SHP Builder would invalidate it when it saves.
I don't think it corrupts the file, because SHP Builder saves Format 80 frames only. So, it usually generates bigger files but it doesn't invalidate the files. Nonetheless, I'm interested to know what does a Loop Frame does and how does it work.
stucuk wrote:
Keeping it separate is less convenient and "floods" a directory with additional files. I prefer clean solutions where possible.
...
I never do. Its also why i have stated from the beginning that everything will be --> optional <--.
When someone chooses this option, the generated file might be invalid in other programs. I don't know how would the option be clear for any user. Many of them simply want to have the convenience of saving the file with the least bureaucracy necessary. So, I prefer to add one additional file in the directory than causing trouble by placing a optional feature that can break other programs, specially with users who doesn't pay attention to optional features. QUICK_EDIT
I have been spending most of my evening going through the Shapeset tool and we have a good idea where and how it writes shape files, even the RLE code it uses (we can provide C pesudo for you if you want to impliment a true compressor or decompressor
With TD SHP's the decode80d which i "nicked"(Hard to nick when it has no license) from SHP Builder seems faulty (As its crashing on a certain TD SHP. I think its crashing with power.shp). I haven't looked at Blade's encoders.cpp and i doubt i will since its GPL (I don't want to make my project tied to "Virus" licenses. ZLib License is the best).
Any extra information/pesudo i can look at would be appreciated.
Banshee wrote:
But in this case, it doesn't make any difference at all.
Technically based on the official SHP's but if i saved a SHP with a flag of 2 and it had zero's in it then SHP Builder would not load it properly. If flag 2 is all the game needs to read it as RLE Zero then SHP Builder would be wrong. Also could be argued about any flag value with the 2nd bit enabled.
Banshee wrote:
I don't think it corrupts the file, because SHP Builder saves Format 80 frames only. So, it usually generates bigger files but it doesn't invalidate the files. Nonetheless, I'm interested to know what does a Loop Frame does and how does it work.
It won't corrupt it. But you lose the loop information.
I am guessing that when it reaches the end of the animation it can loop back to a frame to start again (As in not the first frame). So for example if you had an animation of radar being built and also had frames showing it rotate, you could specify the loop frame at the start of the radar rotating so that once it reached the end it would just show the radar rotating. Pure Speculation on my part as i have yet to find a SHP that actually has a loop.
Banshee wrote:
the generated file might be invalid in other programs
My code is a work in progress, its not a full release application and it can't currently do anything but view shp's and change the radar color. Its not Armageddon yet, there is still time to actually test stuff out. Hell its possible any code i write for saving TD's may be invalid when read by certain applications due to a bug in the code, but thats why stuff gets tested before any proper release.
If the TD Specification doesn't support it(Making the EOF offset point to the true EOF and not the end of the frame data) then it won't happen. There is a difference between the specification not supporting something and a Non-WW application assuming certain values have to be in certain places when they don't.
P.S Irrespective of what i do, since the project is open source even if i added in code to enable CABAL to take over your computers you could easily remove it. _________________ Free Map Editor - Game Requirements - Stucuk.Net QUICK_EDIT
Nonetheless, I'm interested to know what does a Loop Frame does and how does it work.
I am guessing that when it reaches the end of the animation it can loop back to a frame to start again (As in not the first frame). So for example if you had an animation of radar being built and also had frames showing it rotate, you could specify the loop frame at the start of the radar rotating so that once it reached the end it would just show the radar rotating. Pure Speculation on my part as i have yet to find a SHP that actually has a loop.
I suspect that Art.ini turned this feature obsolete at some point, considering that you can already achieve exactly what you described via just Art.ini code (although you can't actually use buildup animations for this purpose). _________________ QUICK_EDIT
The loop frame doesn't actually do anything and is unused by the game as far as I can tell. It looks like TD Shape is descended from the WSA format which is stored as deltas against the previous frame and so a loop frame makes it easy to generate the first frame from the last frame. Only a few shape file have this format in the original, a few radar images in Sole Survivor and all the shapes in wolapi.mix in the 3.0x+ patches for RA.
stucuk, consider this permission to port the encoders.cpp code under a zlib license to your tool. Properly supporting efficient compression is more important than wrangling over licenses. I doubt someone is going to pick it up and build a bunch of closed source products with it anyhow. QUICK_EDIT
Implemented the ISO building type. RA2 buildings with 8 frames will be assumed to be an ISO (Though it can't tell which one). The 4th frame will use the ISO palette.
SHP's can now only be classed as the following types: Unit, Building, Cameo, Animation, Building (IsoSnow), Building (IsoTemperate) and Building (IsoUrban). Types are used for the guess as to the palette as well as some rules like if it has half of its frames as shadow frames or not. If a SHP has a different extension (.sno, .tem, etc) it will use the extension to guess the palette over using the type.
Also implemented the ability to select a palette. The way the system works is that when a SHP is loaded it will guess what it is and use the Type and Game guesses to get the palette. If the user manually selects a palette and you then try to change the type or game it will not change the palette, only the "Rules".
Does anyone know if the first say 12 colours after 0 of a Unit/Building's palette are shadows? RA2 seems to use 1 and 12 for shadows, TD/RA using 4. I noticed that the RA2 units which use 12 have a darker shadow so i am wondering if its a scale where 1 is light shadow and 12 is dark or if they are just special cases.
Blade wrote:
stucuk, consider this permission to port the encoders.cpp code under a zlib license to your tool. Properly supporting efficient compression is more important than wrangling over licenses. I doubt someone is going to pick it up and build a bunch of closed source products with it anyhow.
Does anyone know if the first say 12 colours after 0 of a Unit/Building's palette are shadows? RA2 seems to use 1 and 12 for shadows, TD/RA using 4. I noticed that the RA2 units which use 12 have a darker shadow so i am wondering if its a scale where 1 is light shadow and 12 is dark or if they are just special cases.
No, in TS and RA2 the shadow is drawn by any colour other than #0 as long as its a seperate shadow frame, see E1 Elite's post below (TD/RA use a different palette altogether, one that's preserved as palette.pal in the later games, so I can't say for sure how it behaves). In normal frames, colours #1-#255 are fully opaque, #12 included. I believe they used the solid black colour for unit shadows in RA2 to make them more distinguishable.
This actually reminds me of something I've always felt lacking in SHP Builder: A way to simulate different ambient lighting levels. Basically what such tool would do is allow the user to lighten/darken only those select colours that are affected ingame (see the above link). It could have a simple lighting slider or value input, and an additional tick box to determine the type of the unit (SHP with or without turret) if this is not already determined by the SHP class. This would be very useful for designing unit muzzle anims and to see if there are any stray special-case colours in the wrong place.
Unfortunatelly I don't know if the palette is similarly affected in RA2/YR. _________________ Last edited by Crimsonum on Wed Aug 24, 2016 2:04 pm; edited 1 time in total QUICK_EDIT
#1 if used will look blue, not semi-transparent in normal frames. There is no gradient for
shadows. Though #1 is the designated color for shadows in shadow frames, but game can
handle any color other than #0 as shadow in shadow frames. QUICK_EDIT
Ah yes, I had forgotten about that. I'm too used to the custom of using only colour #1 for shadow frames. Edited my original post. _________________ QUICK_EDIT
(TD/RA use a different palette altogether, one that's preserved as palette.pal in the later games, so I can't say for sure how it behaves).
TD/RA1 don't seem to have any SHP's with shadow frames. They use #4 for shadows.
Crimsonum wrote:
This actually reminds me of something I've always felt lacking in SHP Builder: A way to simulate different ambient lighting levels. Basically what such tool would do is allow the user to lighten/darken only those select colours that are affected ingame (see the above link). It could have a simple lighting slider or value input, and an additional tick box to determine the type of the unit (SHP with or without turret) if this is not already determined by the SHP class. This would be very useful for designing unit muzzle anims and to see if there are any stray special-case colours in the wrong place.
Given the way SHPToolz is coded this shouldn't be hard to implement. So basically just need to make it alter everything but the last 16 palette colours(From what i can tell a custom palette would simulate a turret shp). _________________ Free Map Editor - Game Requirements - Stucuk.Net QUICK_EDIT
Implemented the "Ambient Lighting". I implemented it based on changing the Luminosity of the Palette colours. I don't know if this is the right approach (Though technically it should be the "proper" way to do it). When its set to 0 it will have no effect (Middle of the Trackbar). Currently with a range of -50 to 50. _________________ Free Map Editor - Game Requirements - Stucuk.Net QUICK_EDIT
Haven't checked in game but WW did use HSV for the palette manipulations so highly possible is accurate.
Can you add printing out the Zero? dword too to the debug option?
Apparently at some point ShapeSet had a SHP linking option, the code is still somewhat in the game and ShapeSet just unknown in what state.
I suspect the shps that have data there contain a WWCRC32(just like mixes) or some other of many WWs hash methods checksum of the linked file filename. _________________ Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1
Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite QUICK_EDIT
Does anyone have any recent information on TMP files? I found some information about both RA and TS but Olaf's TS one doesn't go into much specifics (Like how to work out how many tiles are in the file).
Can you add printing out the Zero? dword too to the debug option?
Apparently at some point ShapeSet had a SHP linking option, the code is still somewhat in the game and ShapeSet just unknown in what state.filename.
I can't find any SHP's which are not either TD/RA1 or a TMP file which have a Zero as the first 2 bytes.
E1 Elite wrote:
That won't change the SHP as the palette index would be the same. It is like viewing the SHP
with another palette.
Unless i misunderstood Crimsonum, he seemed to want the ability view a SHP with different ambient lighting. SHP Builder has a lighten/darken tool for pixels. _________________ Free Map Editor - Game Requirements - Stucuk.Net QUICK_EDIT
Does anyone have any recent information on TMP files? I found some information about both RA and TS but Olaf's TS one doesn't go into much specifics (Like how to work out how many tiles are in the file).
Can you add printing out the Zero? dword too to the debug option?
Apparently at some point ShapeSet had a SHP linking option, the code is still somewhat in the game and ShapeSet just unknown in what state.filename.
I can't find any SHP's which are not either TD/RA1 or a TMP file which have a Zero as the first 2 bytes.
I posted WWs OFFICIAL header for the Icon format in the first page already :/
The TD Icon is just the header and the image data afik, haven't looked into it that much.
The Icon format in RA is a new version actually, the header is the same but now in addition to image data it contains OccupyList and OverlapList for each tile and some other bits of information. In TD the info was hardcoded into the exe.
Random fun fact, the Icon format has been around since 1988 with BattleTech: The Crescent Hawk's Inception being the first WW game to use it. That format version was largely IFF specification based.
Not really sure this tool should have Icon support frankly, maybe it should be a separate tool..
What are you talking about.. See the ts shp header i posted in the same post where the Icon(tmp) header is. _________________ Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1
Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite Last edited by tomsons26lv on Thu Aug 25, 2016 2:21 pm; edited 3 times in total QUICK_EDIT
That won't change the SHP as the palette index would be the same. It is like viewing the SHP
with another palette.
Unless i misunderstood Crimsonum, he seemed to want the ability view a SHP with different ambient lighting. SHP Builder has a lighten/darken tool for pixels.
Correct, what you did is exactly what I wanted. Thanks!
I noticed a small error, though: The last colour in the palette (#255) is actually affected by the ambient lighting. So only colours from #240 to #254 should not be affected by the luminosity change. _________________ QUICK_EDIT
That won't change the SHP as the palette index would be the same. It is like viewing the SHP
with another palette.
Unless i misunderstood Crimsonum, he seemed to want the ability view a SHP with different ambient lighting. SHP Builder has a lighten/darken tool for pixels.
Correct, what you did is exactly what I wanted. Thanks!
I noticed a small error, though: The last colour in the palette (#255) is actually affected by the ambient lighting. So only colours from #240 to #254 should not be affected by the luminosity change.
Yea you should do some ini config file for pals to define the indexes that don't get affected by lighting, i highly doubt its just unit*.pals _________________ Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1
Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite QUICK_EDIT
Afaik it's really only the colors #240 to #254 in the unit*.pal's that stay in normal brightness, regardless of the maps Ambient setting.
For other pals it's either all colors get affected or none.
Thus only on the unit*.pal's you can have graphical issues with glowing pixel.
An ini might be only useful if OpenRA (or other engine) implements such a concept in a user definable way and thus this tool trying to stay compatible for such future projects. _________________ SHP Artist of Twisted Insurrection: Nod buildings
Joined: 22 Nov 2010 Location: Iszkaszentgyorgy, Hungary
Posted: Thu Aug 25, 2016 2:36 pm Post subject:
In, OpenRA, any palette with AllowsModifiers: false stay in normal brightness, regardless of lighting. Changing this via mixing lit-unlit colors to a single palette probably won't happen because that harms renderer performance. _________________ "If you didn't get angry and mad and frustrated, that means you don't care about the end result, and are doing something wrong." - Greg Kroah-Hartman
=======================
Past C&C projects: Attacque Supérior (2010-2019); Valiant Shades (2019-2021)
=======================
WeiDU mods: Random Graion Tweaks | Graion's Soundsets
Maintainance: Extra Expanded Enhanced Encounters! | BGEESpawn
Contributions: EE Fixpack | Enhanced Edition Trilogy | DSotSC (Trilogy) | UB_IWD | SotSC & a lot more... QUICK_EDIT
OpenRA has supported traits that modify only a subset of a palette since approximately forever. This is how the water / stealth / blinking light effects work. The "global lighting" modifiers that change the whole palette equally is just a special case of that logic. QUICK_EDIT
At first glance, I thought Crimsonum wanted more than just a view feature as a custom palette
would have given that info on the glowing pixels. If that same logic is available for isotem.pal
or non unit*.pal custom palettes, it would give incorrect info.
If you are looking for more viewing features, some ideas:
- tool could provide a mask on the palette to show which colors are used or not used for
current frame or in the first half of the frames or the whole of the SHP.
- or a selectable palette where user can click on colors to be pinked out to see if the image
is affected.
- animation could have a loop from-to frame numbers to view only a subset of frames looped
- tool could show the compression used alongside the frame counter.
- could allow the background scene to come from an image. It might help in screenshot or
even cameo making.
There could be more complex ones if you get into comparing 2 SHPs or frames.
Hope your palette dropdown extends for longer filenames like unttemmask1ra2bgremap.pal.
And I wouldn't want those palette colors to be saved along with the SHP file as half of the
palette colors would be masked pink color. But instead it would be for viewing with unittem.
As I understand the palette saving with SHP will be optional. QUICK_EDIT
As for the custom background image suggestion, it would be very useful if you could load multiple SHPs for one preview. So you don't have to go ingame to check how it turns out.
Just load the background image (screenshot of a ingame landscape), load the SHPs, set the loop frames for each SHP and click play for a nice preview.
Then a working gif exporter would be great (not this color ruining thing in SHP builder) _________________ SHP Artist of Twisted Insurrection: Nod buildings
You can post new topics in this forum You can reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum You cannot attach files in this forum You can download files in this forum
Outcome of this I seen is, shp (C3) ends up 2.10mb and pcx of same content 1.60mb...
