729752 - Old .pyc files in srcdir shouldn't break the build

Status: RESOLVED DUPLICATE of bug 1188224

(Firefox Build System :: General, defect)

Description

[Jesse Ruderman]

make[8]: Entering directory `/Users/jonathan/mozdev/mozilla-inbound/obj-ff/xpcom/typelib/xpidl'
make[8]: *** No rule to make target `../../../xpcom/idl-parser/xpidllex.py', needed by `libs'.  Stop.

Workaround: rm $srcdir/xpcom/idl-parser/*.pyc (see bug 723861)

Why does the existence of obsolete *.pyc files break the build? Please make configure warn or delete them, or make |make| not fall over, or something.

Comment 1

[Frederic Bezies]

You can add linux to it. Cannot get firefox to be built with the same error on my archlinux.

Comment 2

[Jonathan Kew [:jfkthame]]

(In reply to Jesse Ruderman from comment #0)

> Why does the existence of obsolete *.pyc files break the build? Please make
> configure warn or delete them, or make |make| not fall over, or something.

Yes, it'd be nice to fix this. To be specific, though, I don't think configure should delete them - running configure should not modify the source tree in any way, IMO. And a configure warning would be too easy to overlook.

The correct fix, I think, is to make sure that at |make| time they don't distract |make| (or header.py?) from doing the right thing. But I haven't dug around to understand what all this is really about.

Comment 3

[Bengt-Erik Soderstrom]

The same goes for Seamomkey as well:
(Linux Ubuntu 11.10, gcc 4.6.1)

rm $srcdir/mozilla/xpcom/idl-parser/*.pyc is fixing this bug.

Comment 4

[David Bolter [:davidb] (NeedInfo me for attention)]

Ping. Any |make| gurus out there?

Comment 5

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I don't think we want make automatically changing what's in your srcdir.

Comment 6

[Fred Wenzel [:wenzel]]

FWIW, pyc files are Python build artifacts and should never[tm] matter, so as a Python dev I would not be upset if a build script rm-ed pyc files inside my build dir.

Comment 7

[John Daggett (:jtd)]

(In reply to Kyle Huey [:khuey] ([e-mail]) from comment #5)
> I don't think we want make automatically changing what's in your srcdir.

As stated in the description, make should not fall over like this.  Deleting contents of the src dir doesn't sound like the solution but a solution is needed.

Comment 8

[Gregory Szorc [:gps]]

I think the proper solution would be something along the lines of bug 661908. Having .py files scattered about like we do is just silly and nobody denies that.

Comment 9

[(not currently active) Ted Mielczarek]

AFAIK, the problem from comment 0 occurred because we used to have these (generated) .py files checked-in to the repo, and we moved to having them generated in the objdir. While it's unfortunate that this broke stuff, it's also not a really common scenario.

Comment 10

[John Daggett (:jtd)]

(In reply to Ted Mielczarek [:ted] (away until ~March 7) from comment #9)
> AFAIK, the problem from comment 0 occurred because we used to have these
> (generated) .py files checked-in to the repo, and we moved to having them
> generated in the objdir. While it's unfortunate that this broke stuff, it's
> also not a really common scenario.

Then shouldn't the .pyc files be obsoleted?  I seem to hit this when updating local clones that I haven't updated for a couple months.  Damn annoying...

Comment 11

[John Daggett (:jtd)]

When building on OSX 10.5, I seem to hit this fairly often.  Not sure if this is related but it seems to be correlated with seeing this hg message:

/builds/mozcentral$ hg update
note: possible conflict - build/unix/check_debug_ranges.py was renamed multiple times to:
 build/autoconf/check_debug_ranges.py
 js/src/build/autoconf/check_debug_ranges.py
922 files updated, 0 files merged, 26 files removed, 0 files unresolved

Building on OSX 10.7 I see the same message from hg but I don't run into the problem with .pyc files causing the build to fail.

Comment 12

[(not currently active) Ted Mielczarek]

That's unrelated, FWIW.

Comment 13

[Frank Yan (:fryn)]

Many engineer hours have been wasted hitting this bug. Perhaps, the fix is obvious for the engineers who have knowledge in the compilation process, but for many engineers, it is completely unintuitive what to do once they hit this bug. First, they might try to rm -rf the object directory and do a clean rebuild (as I tried to do), but that doesn't work, so they might resort to doing a clean clone, which takes hours.

Could we get this fixed to avoid more hours of developers' time wasted?

Comment 14

[John Daggett (:jtd)]

Attachment: patch, remove .pyc/.pyo from the .hgignore file

If .pyc/.pyo files are now build into the obj-dir, then it seems like they shouldn't be included in .hgignore.

Comment 15

[Masatoshi Kimura [:emk]]

Comment on attachment 606104 [details] [diff] [review]
patch, remove .pyc/.pyo from the .hgignore file

This patch doesn't look correct to me because .pyc files for non-generated .py files will still be generated in the srcdir.

Do you still hit this bug after changeset https://hg.mozilla.org/mozilla-central/rev/b6627f28b7ec ?

Comment 16

[(not currently active) Ted Mielczarek]

Comment on attachment 606104 [details] [diff] [review]
patch, remove .pyc/.pyo from the .hgignore file

Review of attachment 606104 [details] [diff] [review]:
-----------------------------------------------------------------

They're not, in the general case, only in that one particular case. We still have lots of .py files in the srcdir that wind up with .pyc files next to them.

Someone mentioned on IRC that Python 3.2 has an option to specify an alternate directory for .pyc files, so maybe at some point in the future we can use that.

Comment 17

[Mike Hommey [:glandium] OoO back on Apr 15]

(In reply to Ted Mielczarek [:ted] from comment #16)
> Someone mentioned on IRC that Python 3.2 has an option to specify an
> alternate directory for .pyc files, so maybe at some point in the future we
> can use that.

Alternatively, we could use the -B option, available since python 2.6. which may or may not make things slightly slower.

Comment 19

[Ryan VanderMeulen [:RyanVM]]

Lately, I'm having this issue on the second pass of my pgo builds when I delete all *.pyc files in my tree prior to starting the build.

Comment 20

[Shadowized]

(In reply to Ryan VanderMeulen from comment #19)
> Lately, I'm having this issue on the second pass of my pgo builds when I
> delete all *.pyc files in my tree prior to starting the build.

I'm using pymake and have the same issue right after its supposed to be generating PGO'd code (a bit after the test window closes).

No rule to make target '../../../xpcom/idl-parser/xpidllex.py' needed by ['<comm and-line>', '../../../xpcom/idl-parser/xpidllex.py']

Comment 21

[Edmund Wong (:ewong)]

I remove all the *.pyc files in xpcom/idl-parser.  I remove xpidllex.py and
xpidlyacc.py.  (hg status gives me ?)

Now I get this:

make[9]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom/typelib/xpt/tools'
make[8]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom/typelib/xpt/tools'
make[7]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom/typelib/xpt'
make -C xpidl export
make[7]: Entering directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom/typelib/xpidl'
c:/mozstuff/mozprgs/sm/objdir-sm/mozilla/config/nsinstall.exe -D ../../../dist/sdk/bin/ply
c:/mozstuff/mozprgs/sm/objdir-sm/mozilla/config/nsinstall.exe -m 755 /c/mozstuff/mozprgs/sm/mozilla/other-licenses/ply/ply/__init__.py /c/mozstuff/mozprgs/sm/mozilla/other-licenses/ply/ply/lex.py /c/mozstuff/mozprgs/sm/mozilla/other-licenses/ply/ply/yacc.py ../../../dist/sdk/bin/ply
make[8]: Entering directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom/typelib/xpidl'
make[8]: *** No rule to make target `../../../xpcom/idl-parser/xpidllex.py', needed by `libs'.  Stop.
make[8]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom/typelib/xpidl'
make[7]: *** [export] Error 2
make[7]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom/typelib/xpidl'
make[6]: *** [export] Error 2
make[6]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom/typelib'
make[5]: *** [export] Error 2
make[5]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom'
make[4]: *** [export_tier_platform] Error 2
make[4]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla'
make[3]: *** [tier_platform] Error 2
make[3]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla'
make[2]: *** [default] Error 2
make[2]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla'
make[1]: *** [default] Error 2
make[1]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm'
make: *** [build] Error 2

Comment 22

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I think we should do the -B option.

Comment 23

[David Bolter [:davidb] (NeedInfo me for attention)]

(In reply to Kyle Huey [:khuey] ([e-mail]) from comment #22)
> I think we should do the -B option.

Sounds right. Needs an assignee.

Comment 24

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

It needs performance testing too.

Comment 25

[Robert]

Attachment: build fix for python cache files

This problem occurs in the firefox-13 tarball, and there are no pyc files in the source. This was discussed, and a patch found, on mozilla.dev.apps.firefox.

https://groups.google.com/forum/#!topic/mozilla.dev.apps.firefox/o9FXYPnZu5A

This patch is from that thread. I didn't create it, but I've tested it and it works for me.

Comment 26

[Mike Hommey [:glandium] OoO back on Apr 15]

(In reply to Robert from comment #25)
> Created attachment 635321 [details] [diff] [review]
> build fix for python cache files
> 
> This problem occurs in the firefox-13 tarball, and there are no pyc files in
> the source. This was discussed, and a patch found, on
> mozilla.dev.apps.firefox.
> 
> https://groups.google.com/forum/#!topic/mozilla.dev.apps.firefox/o9FXYPnZu5A
> 
> This patch is from that thread. I didn't create it, but I've tested it and
> it works for me.

Bug 754625 fixed that particular problem.

Comment 27

[Martijn Wargers (dead)]

I just encountered this build error on Windows. How can I circumvent it?

Comment 28

[Martijn Wargers (dead)]

From irc:
[18:59]	<glandium>	mw22: find -name \*.pyc | xargs rm

Comment 29

[Fred Wenzel [:wenzel]]

(In reply to Martijn Wargers [:mw22] (QA - IRC nick: mw22) from comment #28)
> From irc:
> [18:59]	<glandium>	mw22: find -name \*.pyc | xargs rm

That'll barf if you have whitespace anywhere in the results -- you can just use

find -name \*.pyc -delete

Comment 30

[Benoit Jacob [:bjacob] (mostly away)]

I just ran into this. Please fix!

Comment 31

[Ting-Yu Lin [:TYLin] (PDT, UTC-7)]

Is this dup bug 1188224?