Issue 83540 - Add support for multi processing for native W32 dmake
Summary: Add support for multi processing for native W32 dmake
Status: CLOSED FIXED
Alias: None
Product: Build Tools
Classification: Code
Component: dmake (show other issues)
Version: current
Hardware: All Windows, all
: P3 Trivial (vote)
Target Milestone: ---
Assignee: hjs
QA Contact: issues@tools
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-11-11 22:52 UTC by quetschke
Modified: 2013-08-07 15:34 UTC (History)
3 users (show)

See Also:
Issue Type: DEFECT
Latest Confirmation in: ---
Developer Difficulty: ---


Attachments
Patch for dmake (26.81 KB, patch)
2007-11-11 23:00 UTC, quetschke
no flags Details | Diff
Patch for dmake (4.94 KB, patch)
2007-11-12 00:01 UTC, quetschke
no flags Details | Diff
Testcase (355 bytes, patch)
2007-11-26 00:30 UTC, quetschke
no flags Details | Diff
Patch for native W32 dmake (1.00 KB, patch)
2007-11-26 03:36 UTC, quetschke
no flags Details | Diff
Patch for dmake (5.39 KB, patch)
2007-11-28 04:07 UTC, quetschke
no flags Details | Diff
dmake error from "build --all -- -P6" (175 bytes, text/plain)
2007-12-04 17:51 UTC, hjs
no flags Details

Note You need to log in before you can comment on or make changes to this issue.
Description quetschke 2007-11-11 22:52:43 UTC
Allow parallel building with native W32 dmake. Currently the -P# (MAXPROCESS)
switch is not working.
Comment 1 quetschke 2007-11-11 22:57:33 UTC
The following preliminary patch enables the multi process feature for MinGW
builds, the build changes for MS compilers are not ready yet.

With the patch the following makefile builds a1 and a2 in parallel:

D:\w1\cws_src680_dmake412>dmake\dmake.exe -P3 -rf makefile.mk
Start a1
Start a2
End a1
End a2
Build all

- - - start makefile.mk - - -
SHELL:=cmd.exe 
SHELLFLAGS:=/S /C

all : a1 a2
	@echo Build all

a1 :
	@echo Start a1
	@sleep 4
	@echo End a1

a2 :
	@echo Start a2
	@sleep 4
	@echo End a2
- - - end makefile.mk - - -
Comment 2 quetschke 2007-11-11 23:00:30 UTC
Created attachment 49584 [details]
Patch for dmake
Comment 3 quetschke 2007-11-11 23:58:44 UTC
The following patch makes dmake buildable with .NET2003 again. I verified that
my minimal parallel build testcase also works with that version. I hope this
also compiles with MSVC 6.0, but I cannot test that.
Comment 4 quetschke 2007-11-12 00:01:14 UTC
Created attachment 49585 [details]
Patch for dmake
Comment 5 quetschke 2007-11-12 00:12:39 UTC
Both patches committed to CWS dmake412.
Comment 6 quetschke 2007-11-25 17:59:09 UTC
Everything is committed and the simple test case works in parallel mode for
native W32 dmakes -> FIXED.

It would be nice if our testsuite could test MinGW / VC build dmakes, but
currently it can not.
Comment 7 quetschke 2007-11-25 17:59:45 UTC
Please verify.
Comment 8 quetschke 2007-11-26 00:30:01 UTC
Hrmpf, there seems to be a subtle difference that manifests itself only with
a 4NT OOo build. The following attached makefile works fine with dmake411
and 4nt, but fails with dmake412. In a regular cmd.exe shell:

D:\w1\tin_ws2>dmake412.exe -rf dm_test.mk
"Compiler Version 13.10.3077"
dmake412.exe: Executing shell macro: +type testfile $(CFLAGSVERSION_CMD)
awk: (FILENAME=- FNR=1) warning: error writing standard output (Bad file descriptor)
dmake412.exe:  Error code 129, while making 'Shell escape'

This works fine if cmd.exe is used instead of 4nt :(
Comment 9 quetschke 2007-11-26 00:30:47 UTC
Created attachment 49887 [details]
Testcase
Comment 10 quetschke 2007-11-26 03:36:23 UTC
Created attachment 49889 [details]
Patch for native W32 dmake
Comment 11 quetschke 2007-11-28 04:07:01 UTC
Created attachment 49955 [details]
Patch for dmake
Comment 12 quetschke 2007-11-28 14:38:46 UTC
Building OOo is a nice testcase ;)

@ause: As is said on IRC I am still a little bit suspicious about the code in
unix/rmprq.c, but the bug was not using it and to use the code from the single
process only msdos/rmprq.c.

A 4nt build with -P2 ran successful until hitting an expected break in m236 in
canvas/source/directx.

Please verify.
Comment 13 hjs 2007-12-04 17:51:10 UTC
Created attachment 50095 [details]
dmake error from "build --all -- -P6"
Comment 14 hjs 2007-12-04 17:52:39 UTC
from tim e to time the build breaks and i get pages of the error attached. will
try top get some context
Comment 15 hjs 2007-12-04 19:11:41 UTC
wow, 30000 lines of errors before actually stopping ;)
tends to break in extras, but no more information from the log
Comment 16 quetschke 2008-01-08 21:00:26 UTC
@ause: Can you try again? I don't think I fixed it, but I added some diagnostics
and also caught the loop when quitting.
Comment 17 quetschke 2008-02-06 19:37:49 UTC
Some more patches and clean-up changes went in, and ause confirmed that it seems
to work now.

Setting to FIXED, please verify.
Comment 18 hjs 2008-02-25 16:45:44 UTC
it still does :)
Comment 19 hjs 2008-06-05 15:06:05 UTC
.