This is a small document which should explain some basic things about the system used to build the allegro web page. If you have any problems with it or something is unclear, please drop me an email at gradha@users.sourceforge.net.
The following instructions presume you are using a command line Unix-like version of CVS. Most graphical CVS clients behave practically the same and have similar options:
Initial anonymous login and checkout:
cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/alleg login cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/alleg co allegrowww
Initial developer setup and checkout:
export CVS_RSH=ssh cvs -z3 -d:ext:developername@cvs.sourceforge.net:/cvsroot/alleg co allegrowww
Usual update command to retrieve latest changes:
cvs -z3 update -Pd (retrieves new directories and deletes unused ones)
Usual diff command (in case you want to send a diff to somebody):
cvs -z3 diff -u > filename
Usual add command (to add files _and_ directories):
cvs add dirname cvs add filename1 filename2 filename3
Note you have to add dirname before adding dirname/filename1.
Usual remove command (forces deletion of specified files):
cvs remove -f filename1 filename2 filename3
Note you can't remove directories. Leave them empty, and cvs should ignore them.
Usual commit command (use only after a verified update command):
cvs -z3 commit (this opens your editor and waits for your log message) cvs -z3 commit -m "put here your message" (avoids opening the editor)
Note that ALWAYS BEFORE commiting files you should run cvs update to make sure you are commiting changes against the latest cvs version of the files.
Please check the full CVS documentation for details of specific commands.
You need GNU make on your system (other make versions might work, but are untested/unsupported). Under linux you can get it from where you got your distro as a developer package. Under dos/windows you should be able to get it somewhere from http://www.delorie.com/djgpp/.
You need python. The script has to be run with a version 2.x, because it uses a function of the module md5 previously not available, and it is required to check the integrity of the mirrored binary files. Recent python versions should be available at http://www.python.org.
Aparently that is the minimum you need to get the build working. But before you try, the default options rebuild the file external/makedoc.exe. makedoc is Allegro's documentation tool, it is used to generate files like the online documentation api, the faq, the help file, etc. This means that the build process will fail if you don't have gcc or djgpp installed on your machine.
If you can't build makedoc yourself, you will have to create a Makefile.local file with MAKEDOC_DEP = (see below), get a working makedoc binary for your platform and place it in the external dir. If you want to get the compiler, get it from the same place you got GNU make. I can provide you a makedoc binary for DOS or i386/Linux platform (which surprise surprise seems to run at the sourceforge shell).
Ok, so you got the CVS tree of the allegrowww module, with read/write or just read permission. Note that there are a few directories which will mean something:
- src
- This contains the basic layout of the web pages.
- data
- This contains the texts of the web pages, in different languages.
- extra
- This contains the allegro documentation in native ._tx format plus some helper scripts and the makedoc tool.
The build process is started by running GNU make inside the allegrowww directory. By default the make process will put files under the out/ directory. This doesn't exist yet on your hard disk, and if you don't create it, the makefile depend target will do it for you, but you can use this to your advantage and create a symbolic link to another directory, for example a link to an exported Apache folder, so you could test the web pages as if you were really online, but in your own intranet. All the generated files are however plain html, so this is not a requirement and you can use a normal browser to see the result in your hard disk.
The makefile is pretty simple because most of it is generated dynamically through the webmake.py script, and put in the dependency_info file, which will contain all the rules to build the pages. You are not supposed to touch the file Makefile. Instead, you can create Makefile.local and override there the global parameters to your personal taste (see below).
Note that CVS won't contain binary files: zips, tars, and images are not to be commited to the repository, because it does a poor job of keeping binary files. You will find that after the first make run you don't have an out/images directory as well as a few others. To get them, you can use the make mirror command as described in section Mirrored binary files.
This file doesn't exist in the repository, and will be ignored when you commit new files. It is used to allow you personalization of the build process, and most importantly to let you configure it so that it works. Yes, the process as is works under Linux, but will surely fail under Windows or any other platform.
The most important variables you can override are:
- PYTHON_EXEC
- Should point to your python binary. This allows you to have different python versions installed. On windows for example you should do something similar to: PYTHON_EXEC = C:\PYTHON21\PYTHON.EXE. Under linux you could use: PYTHON_EXEC = /usr/local/bin/python2.1.
- STDOUTPUT
- By default is $@ and will work for most people. Some Windows, however, seem to have problems with path forward slashes appearing in $@, and work only with backslashes. To correct this, use: STDOUTPUT = $(subst /,\,$@)
- MAKEDOC_DEP
- Points by default to $(MAKEDOC), which is the makedoc binary. If you don't want to recompile it whenever its source is updated through CVS, set this empty: MAKEDOC_DEP =. This means that you will have to get it from another place and put it in external/makedoc.exe, or the build process will fail at some point during the generation of the documentation from the Allegro sources.
- GENERATE_LANGUAGES
- A string made up of language codes separated by colons. This variable is used to determine which langcode pages will be created in the out directory.
- GENERATE_ONLINE_DOCS
- Just like the previous value, indicates for which langcodes should be generated the external online documentation.
Apart from those you will hardly need to touch more variables, but just in case you can take a look at the beginning of the Makefile.
This file works as Makefile.local: you won't get it from the repository. You will get however allegro.ini.example which is a template. Check the template and use it only if you need to specify some values other than the default ones, because the values provided in the example are hardcoded in the python script.
external_languages and external_online_docs are similar to the Makefile.local GENERATE_LANGUAGES and GENERATE_ONLINE_DOCS variables. If you reduce in Makefile.local the range of generated files, you will have missing pages accesible through the top language bar at each page. Setting external_languages to the languages you left out in GENERATE_LANGUAGES will transform those broken local links to external links set by external_language_link. This is used for the Demon mirror which only holds the english version of the web page and documentation in order to save web space (it's Shawn's account, you know personal web pages usually don't have much space available).
The external_online_docs variable is the same story but with the online docs which are generated through makedoc and put at out/onlinedocs/xx/ where xx is a language code.
As you will see, there's a little problem with these two variables. What happens if a new translator adds to the global Makefile a new language like it for italian? The result will be that unmaintained sites will generate online docs and local broken links for that langcode. Therefore, in the future these variables will be changed to local_languages and local_online_docs, which will be negations of the actual ones, and thus will avoid that future langcode additions generate bad html code.
Finally, the allowed_includes variable is a colon separated string of keywords. These keywords appear as optional arguments to include tags in the form <include-(data|source)[-keyword]=filename>. If the script finds an include tag with a keyword option, and this keyword doesn't exist in the allowed_includes variable, the script will happily ignore the tag and continue processing. Such behaviour allows to customize html output in a per site basis, and is used for example at the end of the src/menu_bar file to show a "Hosted by Sourceforge" logo, which will obviously appear only at the Sourceforge mirror.
Previously there was a kill_warnings.txt file, but now there isn't, it wasn't used, and it wasn't useful at all.
The build process consists of make running the python script to generate the output page. Also, the build process as is when you get it from CVS is definitely oriented towards Unix users. In order to make it work under Windows, you will have to follow the advice given in the sections 4 and 5 of this document.
The first time you run make, the scripts will generate a dependency_info file. It contains information about which source files depend on which localizations for every langcode html version. It will also contain the build rules for each langocde file. Sometimes you will have to regenerate the dependency information, and this happens when there a global change in the Makefile, like a new langcode, or you have just added a new localization file for one of the english versions found in the data directory. In such cases, running 'make dep' will erase the file, and the next time you run make, it will be regenerated.
If you have a mirror and want to mirror the web site, your usual cron job should contain something very similar to the following:
cd directory_where_you_keep_the_allegrowww_module cvs update -Pd make mirror && make dep && make && make
make mirror is explained in chapter Mirrored binary files. You will notice something strange in that script. First of all, the dependency_info file is rebuilt even when no changes have been done to CVS. Then, you will see that there are two make commands in a row. That is because the external files found at data/external need a special treatement. The first time you run make, files will be created in those directory, but since the dependency_info file was generated when they still didn't exist, source files might not be updated correctly yet.
To correct this, the build rule for files in the data/external directory erases manually the dependency_info file, meaning that the next time make is run, correct dependency information will be generated. Of course, if the CVS has not touched any of those files, the second make will just say "nothing to do", so it doesn't waste too much CPU time anyway.
Note that there's a src/force_update file. All web output depends on this file, and it's there exactly to force a whole web update when for example the webmake.py script changes its output. Since the file will be updated through CVS, touching its timestamp is not enough, you have to modify its contents to force a whole web update. Adding a character to it will be enough. In short, you can ignore this file if you are not developing the webmake.py and related web generation scripts.
If you are interested only in maintaining a mirror of the web page, and have a Unix environment, you can use scripts/update-website.py. This script runs the cvs/mirror/make commands in the correct order and keeps the local copy updated. In fact, it is used on SourceForge to update the web from the CVS automatically. If you choose to go this path what you have to do is:
- Check out the allegrowww module.
- Configure your allegro.ini and Makefile.local.
- Put scripts/update-website.py inside a cron job.
- Profit.
The script is by default totally quiet, as you don't want successful jobs to be reported through cron's email messages. If you do, there's a switch you can turn on to get the full output of all updates. The script is command line friendly, run it once to get info on all parameters. You shouldn't need to run the script more than once a day, as the Allegro web page is fairly static with few updates every month.
The previous script is not enough to have the Allegro page correctly updated at the SourceForge mirror. But only because there is some trickery to make part of the Allegro web page look dynamic. It is the CVS snapshot file, which on SourceForge contains a listing of snapshot files and on other mirrors it contains a reference to the SourceForge dynamic page. See the first comment of scripts/sourceforge/update_cvs_snapshots.py to see how this is done. This script also trims the number of available snapshots to ten.
Another thing that is good to do is change the permissions and group of files generated by webmake.py. The group should be alleg, and the permissions 664.
Both of these tasks, as well as running the update-website.py script are handled by a common cron wrapper found in scripts/sourceforge/update_web.sh. This wrapper only needs the path to the root allegrowww checkout to work. Only the webmaster needs to take care of this cron:
# This updates the Allegro web page. More info at: # http://cvs.sourceforge.net/viewcvs.py/alleg/allegrowww/docs/sf_cron_update_web.txt?view=auto 27 19 * * * allegrowww/scripts/sourceforge/update_web.sh allegrowww
Usually $HOME/allegrowww will be a link pointing to /home/groups/a/al/alleg/cvs_allegrowww_checkout/, so you don't have to type a lot. Extra notes are stored in sf_cron_update_web.txt.
There are also two periodic maintentance scripts run for translators: one tries to show if a translation has been orphaned due to lack of work by the translator. The other shows which text chunks are missing translation for non English languages. Again, the webmaster needs to take care of these crons:
# This notifies a mailing list if a web translation is stale. More info at: # http://cvs.sourceforge.net/viewcvs.py/alleg/allegrowww/docs/sf_cron_author_statistics_update.txt?view=auto 13 11 1 * * $SHELL allegrowww/scripts/author_statistics_update.sh allegrowww # This notifies a mailing list which text chunks haven't been translated yet. More info at: # http://cvs.sourceforge.net/viewcvs.py/alleg/allegrowww/docs/sf_cron_translation_stat.txt?view=auto 05 3 * * sat allegrowww/scripts/sourceforge/translation_stat.sh allegrowww
You've got used to the CVS and start to understand the dependencies: the src directory contains plain layout files. The data directory is divided in several dirs, each of them with the name of one file in the src directory. Again, each of these will have at least an en file, and possibly es, fr, etc, files. These contain the plain text localizations. Note that unlike the src directory, the files under data directory must not have any <include-whatever> tags, because they are meant to contain the final text.
Those two letter extensions are language codes identifying each translation's texts. If you want to contribute the it (italian) language, you will be adding it files everywhere with the translation of the en files.
This is not enough: you will have to modify src/top_header to include the new language link in the top part of each generated html file. This will ensure that the end user can always switch to the same page in another language following that link.
Finally, you have to add some dependency information to the main Makefile to generate the new files. Just add your new language code to the variable GENERATE_LANGUAGES, and the next time you build the dependency information, language pages will be generated. You have to add your language in the form <langcode>,<charset>, where charset is the character set that should be specified in every file's header meta tag. Example for Korean: kr,euc-kr. Omitting the ,<charset> part will presume iso-8859-1 charset.
Now running make dep will generate the new files of the added language code. Note that you may generate them without having any translated files: the webmake.py script will put en texts wherever your language texts are missing, providing a fallback replacement, and notifying you with a warning during the generation. This will allow the english version to evolve without having to wait for the translation files.
Yet another detail you may want to customize is the date timestap printed at the foot of each web page. By default it will show an English date, but you can easily localize the month name. Open data/localize_time/language_code and put in the first line the twelve month names of the year separated by spaces. The second line, which can be optionally empty, indicates the string formatting of the date. The formatting is done using python variable substitution: %(variable_name)[formatting]type. The variables can be: daynumber, monthname, year, hour and minute. formatting is optional and can be a C-like printf formatting string. type will be d for numbers and s for strings. Example for Spanish string:
el %(daynumber)d de %(monthname)s del %(year)d a las %(hour)02d:%(minute)02d (UTC).
Also, please add yourself to the src/webmasters.html file for completeness.
Remember that after you add/remove files you have to run manually make depend, or the new files/changes won't get noticed by the Makefile. If you modify files of the data or src directories and nothing happens please make depend and try again, check your clock/time, etc.
Your language files are meant to be encoded in the same encoding you will be using in the html output: if you are using iso-8859-2 for html output, the localization files should be in iso-8859-2.
IMPORTANT NOTE: Please don't modify src/top_header (to include your language) and Makefile with your language code until a good part of the web is translated. Otherwise, the web will be purely the same like the english version and end users will be disappointed. Use Makefile.local to test locally your translation until you have a good portion of the web page translated to your language. Of course, you are free to release them whenever you want, it all depends on how much free time you can spare for translations :)
There are several pages which are autogenerated from the original Allegro documentation. Effort has been put so that writting once an external/langcode/*._tx file will generate automatically the html version through makedoc and a helper python script which changes slightly the html code. This also means that copying the ._tx files over will update the web page automatically. Nice, eh?
The generated files will be put under data/external/langcode. Those files should not be modified manually or commited, since they are automatic.
There's only one caveat: due to the system used to generate automatically the web dependencies, it might be necessary at times to run make twice. Running make once may work too, but it could work bad, so just to make sure run make twice after regenerating the dependencies. Under Unix systems it's as easy as make && make. The second make won't do anything if there's nothing to do, so it won't slow you down very much.
If you want to provide translations of the external files, create your langcode directory under external, and put there the appropiate xxx._tx translated files. You may want to take a look at the Spanish version to get an idea on how it all works. Remember to generate and commit the nothing.txt file under data/external/langcode/ so that the directory will exist for other people too.
The external Allegro API is also generated from the external directory, but it is put directly at out/onlinedocs/xx/.
You will also notice that having external files generated through makedoc.c requires you to have a working GCC compiler. This can be a problem, like at SourceForge, where the compiler is not available. In such cases, create your Makefile.local with the MAKEFILE_DEP variable set empty, and source updates to makedoc.c won't bother you. However, you will have to manually get makedoc.exe put into the external directory, possibly (cross)compiling at home and uploading the binary to the correct place so that the build of the web pages can proceed.
While CVS and the scripts take care of all the text files, what happens with binary files? Putting them under CVS is a perverted waste, since CVS is not good for binary files. So how to mirror them? With yet another python script (weren't you expecting it?).
The mirror scripts are called running either make mirror or make fake-mirror. Don't run the first one until you understand what is going to happen, and for this, you can use the 'fake-mirror', because it will show you what it's going to do.
Basically, external/binary-files.txt is placed under CVS control, and this file is a crude python file which is imported by the mirror script (note that since it's true python code, wrong syntax will make all the mirror process fail, so be sure to run fake-mirror at least once before commiting to CVS. Another solution is to write a proper config file...). From this file you can call two commands, mirror_files.append and delete.append.
The first is used to register a binary file, and the second is used to delete files which don't have to me mirrored any more. The parameter to mirror_files is a tuple (ie: a comma separated list enclosed in parenthesis). The first element of the list is the 'nick' used from the web pages' source. Usually it will be the same like the filename path, but it doesn't have to, so you can make it shorter if it doesn't clash with other entries.
The second is the URL from where the script can get the file if it has to. The third is the size of the file, the fourth is a md5 hash. You can specify -1 for the length and an empty string for the hash, but you will loose the integrity checks (very needed if an upload is interrupted). The fifth element is the path relative to the out directory where the file will be placed, and the last one is a bracket enclosed string list, which can contain the following keywords: all, none, high, medium, low, no-sourceforge.
all mirrors the file everywhere, none deletes it from all mirrors, and high, medium and low indicate the importance of the file. Usually a mirror will specify in it's allegro.ini config file the levels of mirroring. Finally, no-sourceforge is a special keyword used to avoid the SF mirror enter in an endless loop. Usually this is not needed, but just in case...
Running fake-mirror will tell all the files which will be downloaded or deleted depending on your allegro.ini. You can interrupt both fake-mirror and mirror with Ctrl+C, though the script doesn't know how to resume downloads, so the file which was interrupted will be redownloaded again.
Finally, note that the generation of the web page depends on the existence of the binary files: if you run make before downloading any files, all the links will point to the remote URL. If you make mirror you will have to rebuild again the html files so that their code points to the local file. That's why make mirror is the first command to run after cvs update.
Don't be fooled by the webmasters.html file where it says that Grzegorz Adam Hankiewicz is the main webmaster. In the real world, I will be probably buried under lots of real life work and will be unable to keep up always with each small update. CVS allows anybody with write access to change any file, which means that if I happen to dissapear, somebody can just substitute me.
Apart from the web translators, allegro developers can also update the web without previous notification if needed: they already have write access, so the only thing they have to do is get the allegrowww module, make some changes adding a few files to the data directory and let others translate that to the rest of the available languages.
So don't be shy, if you want to make some changes fast, don't even mail me and modify the web page at will, even in the worst case things can be undone. And web translators will be notified automatically through the allegrowww-commits mailing list, so they will also know what to do.
While the allegrowww-commits mailing list is for CVS commits, I guess everybody can post to it manually to ask something to the whole group, meaning that if you want to make a change and you are unsure about what to do, somebody on the list could have an idea.
I used python for the scripts because I feel very comfortable programming with it. But the language doesn't matter: if you wish to have the parser in C, C++, Java or whatever, just do it. Provide your version and if it works and is better to use than the actual version, it will be used instead.
If there are problems, you can try to fix them yourself or mail me to check what's happening. There shouldn't be any, however, the build process seems to work without problem under DOS and Linux. But gremlins exist...
The following items are roughly a guide to release new WIPs on the web page and have it working correctly. A little bit schematic, but will be improved in the future:
Add a news item in the src/index.html file. Move the oldest one to the src/old_news.html. The former should contain only 3 news items.
Add the corresponding tags in data/index/index.en. You have there the tags of a previous wip release announcement, so you can basically copy it, change the numbers and rewrite the small changelog. To have a good measure of the length the small changelog should have, trim it to no more than 600 characters (without HTML tags). You will surely leave important changes, but that's the point of having a true changelog linked for too curious users!
Update external/en/*._tx with *._tx files from the recently released Allegro package only if this package is a later version than the current one. This means you can ignore stable versions if there's a more advanced WIP version. Don't forget about the files in the build subdirectory.
Update the makedoc.exe binary at the SourceForge shell account updating the web page. This is currently at /home/groups/a/al/alleg/cvs_allegrowww_checkout/external/makedoc.exe. There's no need to do this step if makedoc didn't change. If you do, remember that you have to update the makedoc source in the external folder too, for those mirrors that are able to build it.
Update src/wip.html with correct link numbers, both the defines at the beginning and the later links to files. Note that the real information about download links is stored in external/binary-files.txt, which you will have to update correctly. This usually involves duplicating the entries of the last version and refilling them with the new data. The old entries should be added to the end of the binary-files.txt, in the delete section.
Open src/api.html and change the version of the online documentation.
Optionally generate documentation binaries in pdf and ps.gz format. You will have to upload these to the SourceForge web page before adding them to binary-files.txt or put them on your web site and make sure they get fetched by SourceForge, and then reupdate the binary-files.txt file to point to the SourceForge web page.
Add to the end of src/old.html the new diff link.
Add the tag release if necessary to src/cvs.html.
Commit and check some hours later the web page. See Available mirrors on how/who updates the mirrors.
Announce on http://www.allegro.cc/.
Announce on http://freshmeat.net/ (needs a free user account).
Update allegro entry in the free software directory at http://www.gnu.org/directory/allegro.html
To do so, there are two ways:
provide Janet Casey (jcasey@gnu.org) the new information. (note to self: depending on humans is bad)
check out the entries of the directory from anonymous cvs:
export CVS_RSH="ssh" cvs -d:ext:anoncvs@savannah.gnu.org:/webcvs/directory login cvs -z3 -d:ext:anoncvs@savannah.gnu.org:/webcvs/directory co directory
Then update allegro.txt (note it's in latin1) and send the diff to <bug-directory@gnu.org>.
Scrap all this pile of junk. It is too complicated to make a web change, so it scares everybody other than Grzegorz to change the web. Replace by something more shiny and easy to use.
If you feel something has been left out, don't hesitate to tell me, or to add it directly to the todo.txt file.
At the moment of writting this, three mirrors exist:
All three mirrors are automated now. The SourceForge mirror is updated by a cron script in my shell account (gradha). Every three hours it updates a local anonymous copy of the allegrowww module, and rebuilds modified pages to the web server (which is in another directory at the same server). The only problem with this setup is that the SourceForge shell account doesn't provide gcc, so if the makedoc tool changed, somebody would have to compile it on a linux machine and modify the binary after testing that it runs. This is not so critical, though, because the makedoc format seems to be very flexible and backwards compatible. The three hours period is a little bit long, but that's to avoid abusing the server. There's a little bit of trickery in some script to update the cvs snapshots web.
The Sunsite and Talula mirrors can be updated only through ftp. I've set up a cron job at my personal home machine, which gets the allegrowww module and then, with sitecopy, updates both pages. If for some reason the scripts breaks or the server vanishes, you will have to get ftp password from Shawn Hargreaves and the Sunsite staff in order to be able to update those hosts manually. They are fully automated, and should require no manual maintenance unless something breaks.
The SourceForge server should be considered the primary server, all files are there, and downloads link to SourceForge's download mirrors. The demon mirror only hosts English html, all binary links point to SourceForge to avoid bandwith usage and because Shawn's quota is around 20 Mb, 10 of which are already used. The sunsite server is a mostly complete mirror of SourceForge, it's not complete because it has a disk quota of 50MB, which was exceeded sometime in december 2002, so it was changed to host all binary files except those in the low category.
The file src/mirrors.html contains a list of DJGPP mirrors with direct links to the Allegro directory. This file is updated by scripts/update-mirror-list.py. This script has to be run manually by the webmaster, the script will get an updated list of the mirror list, parse it and regenerate the specified file. Example:
python scripts/update-mirror-list.py -f src/mirrors.html
Now doing a cvs diff will show if there were changes, which should be commited. The idea is that the webmaster automates the execution of the script. The script is written in such a way that the minimum failure will explode with a beautiful log of fireworks. Should the webmaster not update the file, whack him.
In the spirit of keeping web rot to a minimum, the following policy will be enforced:
If a translator has not updated the web in a year:
- The webmasters.html page shows that the language translation is no longer maintained.
- The developer assigned to this task is removed from Sourceforge's group unless he is doing work on other modules of the Allegro project.
- At the top of each page is slapped a "this language is not maintained any more" with a link to the webmasters page.
If a language has not been updated in five years:
- It is removed from the web. Previous work is kept under CVS, of course, in the hope that somebody will come and bring it up to date.
Finally, people who want to contribute a new language have a three month limit of time to get used to the CVS and start commiting changes. Once this time has expired, their commit rights will be removed from Sourceforge and the translator position will be vacant again.
Anybody with admin rights on the Sourceforge project can enforce this policy (removal of users). At least two people should be allowed to do this, the main webmaster and the Allegro dictator.
This document should be available from Sourceforge's CVS allegrowww module in the allegro project. The HTTP link is: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/alleg/allegrowww/instructions.txt.
If you are reading the HTML version, this was generated with reStructuredText, which you can get from http://docutils.sourceforge.net/. If you are reading the txt version, well, that is what the strange markup is for, generating the HTML version.
This document is not automatically generated for several reasons: