OpenApi docs/specs

The filters are not displayed in the OpenApi docs. I would still find it a useful extension. What do you think?

It would certanily be nice if the options to combine existing filters with and, or and not would be displayed in the OpenApi docs!
I was led to believe that the OpenApi docs only support a flat list of keys that activate the filters and types of the values that can be passed to them. But i must admit, i did not really look into it. So maybe OpenApi supports more and can actually express the combination of the existing filters woth and, or and not in a tree of indefinite depth. If you have an idea about how to do that, please explain!
Or maybe a flat list of a configurable number of layers combined with the specs of each of the existing filters would do? But if there are many filters, wouldn't that lead to a very long list? How to keep that within usable limits?

We translate the schema to typescript types using https://www.npmjs.com/package/openapi-typescript. For that purpose a wildcard for any additional "and", "or" and "not" would be sufficient for the beginning. Like that the consuming client knows after the FilterLogic filter possibilities.

Hi,
I do not have the time right now to look into it. Ik hope next week is less busy so that i can take a look.
Thanks for now, Henk.

Flattening out the combinations of logic and filters, like you do under
"Another solution" would be relatively easy but limiting the logic to those
specified would be harder and goes against the idea of allowing the client
to specify the logic. Something like this is discussed on Filters RFC
and here. But not limiting the logic would necessarily leave most options undocumented. Could still be usefull though because Swagger Ui would understand the part that IS documented.
I do not understand what you mean by a wildcard, a description like
{ [key: string]: unknown }
does not seem valid to me.
Reading swagger.io/docs my best guess would be the following:
parameters: [
	(..),
	{
		name: "and",
		in: "query",
		description: "Combines the nested filter specs through AND"
		required: false,
		schema: {
			type: "object"
		}
		style: deepObject
		explode: false,
		allowEmptyValue: true
	},
	{
		name: "and",
		in: "query",
		description: "Combines the nested filter specs through AND"
		required: false,
		schema: {
			type: "object"
		}
		style: deepObject
		explode: false,
		allowEmptyValue: false
	},
	{
		name: "and",
		in: "query",
		description: "Combines the nested filter specs through AND"
		required: false,
		schema: {
			type: "object"
		}
		style: deepObject
		explode: false,
		allowEmptyValue: false
	},
}

Objects additionalProperties by default is true, but i am not sure type "object" itself would be valid.
Instead one could type and, or and not as "#/components/schemas/LogicalFilters" defined as:
parameters: [
	(..),
	{
		name: "and",
		in: "query",
		description: "Combines the nested filter specs through AND"
		required: false,
		schema: {
			{ $ref: "#/components/schemas/LogicalFilters" }
		},
		style: deepObject,
		explode: false,
		allowEmptyValue: true
	},
	{
		name: "and",
		in: "query",
		description: "Combines the nested filter specs through AND"
		required: false,
		schema: {
			{ $ref: "#/components/schemas/LogicalFilters" }
		},
		style: deepObject,
		explode: false,
		allowEmptyValue: false
	},
	{
		name: "and",
		in: "query",
		description: "Combines the nested filter specs through AND"
		required: false,
		schema: {
			{ $ref: "#/components/schemas/LogicalFilters" }
		},
		style: deepObject,
		explode: false,
		allowEmptyValue: false
	},
}

LogicalFilters: {
	type: "object",
	description: "Recursive logical combination with the filter specs as additional properties",
	properties: {
		and: {
			nullable: true,
			description: "Combines the nested filter specs through AND",
			anyOf: [
				{ $ref: "#/components/schemas/LogicalFilters" }
			]
		},
		or: {
			nullable: true,
			description: "Combines the nested filter specs through OR",
			anyOf: [
				{ $ref: "#/components/schemas/LogicalFilters" }
			]
		},
		not: {
			nullable: true,
			description: "Negates the nested filter specs",
			anyOf: [
				{ $ref: "#/components/schemas/LogicalFilters" }
			]
		}
	}
}

Or one could make the type specific to the entity and its filters. For example the
Employee entity
from MetaClass Tutorial Api Platform only has one filter that reacts to the parameter "search":
Employee.logical-filter: {
	type: "object",
	description: "Recursive logical combination with filter specs"
	properties: {
		and: {
			nullable: true,
			description: "Combines the underlying filter specs through AND",
			anyOf: [
				{ $ref: "#/components/schemas/Employee.logical-filter" }
			]
		},
		or: {
			nullable: true,
			description: "Combines the underlying filter specs through OR",
			anyOf: [
				{ $ref: "#/components/schemas/Employee.logical-filter" }
			]
		},
		not: {
			nullable: true,
			description: "Negates the underlying filter specs",
			anyOf: [
				{ $ref: "#/components/schemas/Employee.logical-filter" }
			]
		},
		search: {
			nullable: true,
			description: "Selects entities where each search term is found somewhere in at least one of the specified properties",
			type: "string | array",
			items: {
				type: "string"
			}
		}
	}
}

(I i not sure about the type "string | array", maybe i should use oneOf instead)


Fix checkpoint calculation not excluding dropped segment

Fix #58
/kind bug
Signed-off-by: Congqi Xia<EMAIL_ADDRESS>

/lgtm


Apollo Provider (Front end) / Lint / Vite Proxy

Setup vite proxy; fixed eslint so it will autofix after save of current file (needs change in settings.json - vscode- and I had to take out airbnb); added apollo prov to main.jsx/app.jsx; created PLACEHOLDERs for- Components (folder) & Utils (client side-entire contents)- also user query (placeholder).
Airbnb defaulted double quotes 🥸🤖🤠. I made sure we still had access to login connecting on localhost for apollo p. Gonna try to start work on socket.io.

🤘


fix completed multiupload lost data

If there are putObjectPart requests with the same uploadId during completeMultiPart, it can result in data loss. putObjectPart requests might be due to timeout retries.
What problem are we solving?
S3 completeMultiUpload makes data lost.
If there are putObjectPart requests with the same uploadId during completeMultiPart, it can result in data loss. putObjectPart requests might be due to timeout retries.
How are we solving the problem?
Add an uuid id to the part object name to avoid covering, if this partId has retry requests.
How is the PR tested?

I use aws go client SDK to simulate multi-upload, when doing completedMultiUpload to loop putObjectPart one part simultaneously.
read the object with readDeleted=false to check whether it has data lost.

Checks

[ ] I have added unit tests if possible.
[ ] I will add related wiki document changes and link to this PR after merging.

Who will fix the failed test now?

@kungf please fix the s3 tests

@chrislusf ok

what is the history of the test_multipart_resend_first_finishes_last?

def test_multipart_resend_first_finishes_last():
    bucket_name = get_new_bucket()
    client = get_client()
    key_name = "mymultipart"

    response = client.create_multipart_upload(Bucket=bucket_name, Key=key_name)
    upload_id = response['UploadId']

    #file_size = 8*1024*1024
    file_size = 8

    counter = Counter(0)
    # upload_part might read multiple times from the object
    # first time when it calculates md5, second time when it writes data
    # out. We want to interject only on the last time, but we can't be
    # sure how many times it's going to read, so let's have a test run
    # and count the number of reads

    fp_dry_run = FakeWriteFile(file_size, 'C',
        lambda: counter.inc()
        )

    parts = []

    response = client.upload_part(UploadId=upload_id, Bucket=bucket_name, Key=key_name, PartNumber=1, Body=fp_dry_run)

    parts.append({'ETag': response['ETag'].strip('"'), 'PartNumber': 1})
    client.complete_multipart_upload(Bucket=bucket_name, Key=key_name, UploadId=upload_id, MultipartUpload={'Parts': parts})

    client.delete_object(Bucket=bucket_name, Key=key_name)

    # clear parts
    parts[:] = []

    # ok, now for the actual test
    fp_b = FakeWriteFile(file_size, 'B')
    def upload_fp_b():
        response = client.upload_part(UploadId=upload_id, Bucket=bucket_name, Key=key_name, Body=fp_b, PartNumber=1)
        parts.append({'ETag': response['ETag'].strip('"'), 'PartNumber': 1})

    action = ActionOnCount(counter.val, lambda: upload_fp_b())

    response = client.create_multipart_upload(Bucket=bucket_name, Key=key_name)
    upload_id = response['UploadId']

    fp_a = FakeWriteFile(file_size, 'A',
        lambda: action.trigger()
        )

    response = client.upload_part(UploadId=upload_id, Bucket=bucket_name, Key=key_name, PartNumber=1, Body=fp_a)

    parts.append({'ETag': response['ETag'].strip('"'), 'PartNumber': 1})
    client.complete_multipart_upload(Bucket=bucket_name, Key=key_name, UploadId=upload_id, MultipartUpload={'Parts': parts})

    _verify_atomic_key_data(bucket_name, key_name, file_size, 'A')

@kungf You have fixed a special case of the test, and not the general case when the last loaded part number ends up in the final file.

hi, @kmlebedev I think the test_multipart_resend_first_finishes_last test case is unreasonable, for uncertain input(two part 1) from the client, an error should be returned。 We should not speculate on which option to use, as our guess may not necessarily be what the user wants. We should return an error and let the user confirm their input.
so, how about deleting this test case, and adding a PR to return ErrInvalidPart in scenarios involving the use of the same part?

Let's leave the test, since s3 clients are different and why give an error if you can return the expected result.
https://github.com/seaweedfs/seaweedfs/pull/5471

hi, @kmlebedev which version of s3test seaweedfs use?


[$500] Web -  missing translations on additional details page in wallet enablement flow

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: <IP_ADDRESS>
Reproducible in staging?: n/a
Reproducible in production?: n/a
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: @kevinksullivan
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1697480656854849
Action Performed:

User A: login, navigate to Settings > Wallet
User A: add personal bank account
User B: login, select Request money
User B: enter User A as the person you're requesting from
User A: Navigate to IOU from User B
User A: select Pay with Expensify

Expected Result:
There should be translations for every field.

Zip code: código postal
Personal address: Dirección personal

Actual Result:
Personal address and zip code both show "MISSING TRANSLATION" on expensify.com emails, to call out that we are missing a translation (https://github.com/Expensify/App/issues/27759 for more details)
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01df8c2e9cdd1c685b
Upwork Job ID:<PHONE_NUMBER>895460864
Last Price Increase: 2023-10-16

looking for translation
https://expensify.slack.com/archives/C21FRDWCV/p1697498156628909

Updated the OP. the problem is that we are missing translations, and not that MISSING TRANSLATION is shown on expensify accounts.

@kevinksullivan as @ishpaul777 mentioned, the problem is that we provide already translated text to the translate method again, we should make sure we are not passing this through the translation twice

@tienifr @0xmiroslav This has popped up in couple of flows in the app, for example here too https://github.com/Expensify/App/pull/28907 Could we please fix all these occurences here and try to fix this as generically as possible for Form component? thanks!

honestly i don't understand how this is a new issue and not a regression caused by that PR but 🤷♂

@marcaaron I can take it over if you prefer.
I dont think its a regression because that PR just surfaced this bad practice which should be fixed imho.

It looks like a bug. I thought all the server errors show MISSING TRANSLATION? Why do we want this? 😂

PR is taking a LONG time, removed Reviewing here so it'll go overdue, bumped to weekly too.  @tienifr can you provide an update? (also commented in the PR).  Thx

Why is this taking so long?

Can we please reassign to a new contribuotr if @tienifr is unable to finish this immediately?

@0xmiroslav I think this is waiting on review - could you take a look ASAP?

@peterdbarkerUK , per this chat in #contributor-plus, I've reassigned to @situchan
https://expensify.slack.com/archives/C02NK2DQWUX/p1706043173197049

Guessing we're also putting this on hold @mallenexpensify , in favour of the Missing Translation PR?

Not overdue, @tienifr is working on the PR

Not overdue

PR was merged yesterday!!!

Not overdue

Bump @situchan to complete the checklist please

I made a new job + here is a payment summary:

Contributor: @tienifr $1000
C+: @situchan $1000

Upworks job

I already have offer accepted here
This doesn't require BZ checklist as no way to reproduce on non-Expensify accounts.

@situchan I'm aware that you've accepted the offer, but that was for the old posting and it's not been closed by Upworks:

I don't have a way to pay you on that job, so I created a new, active job.

ok no problem. Accepted new offer

Paid and job closed. Waiting on @situchan to complete checklist

@jliexpensify here's my feedback on checklist - https://github.com/Expensify/App/issues/29712#issuecomment-1951680561

Cool, thanks! Closing!


Can't click on things with various Firefox versions and Linux distros

On the 16th (the day after 2.5.1 was released) iD stopped working properly for me.
Most functionality works.  Background from Bing displays and different backgrounds can be selected. Objects (points/lines/areas) display, labels display.  Map can be slid.  Map can be zoomed with scrollwheel or right-hand controls.  All the right-hand controls work.
Some functionality doesn't work.  The spinny thing never stops spinning.  I left it for hours and it doesn't stop.  At one point, after around 30 minutes, a strip of red along the bottom complained about being unable to connect to API (or something like that), but I can't reproduce that one and it may have been a temporary connectivity issue.  That's a triviality, but there's worse...
Big problems: I can't select objects to edit them.  The top controls (point/line/area) don't operate so I can't add objects.  It's tantalizingly close to being usable without actually being usable.  I can see everything, I just can't add/modify objects.
I've tried this with Firefox 52.4.0 (64-bit) on CentOS 6.8 and on CentOS 7.3.  I even got hold of a laptop with Windows 10 (ugh) and used Edge (double-ugh) and had the same problem.
For a while I thought it might be bit of javascript being loaded from a different server from the rest of it and I was seeing a routeing problem with my ISP or some auto-blocking/tarpitting on a server taking out a block of IPs I happen to be on.  But then...
Today, fumbling with the touchpad on an unfamiliar laptop, I somehow accidentally triggered the help walkthrough (don't ask me how I did that without noticing, I don't know, I only know I hate touchpads).  Clicked to the last bits of the walkthrough to get out of it (I wish "start editing" really went straight to editing instead of launching a mini-walkthrough, or there was a "I came here by mistake, get me outa here right now" button), and when I finally escaped the walkthrough then iD started working properly (the spinny thing continues spinning, but I can edit/add objects).  I've reproduced this on my main computer, so it's consistent.
Note that this fix doesn't survive logging out/back in.

Wow there's a lot to process in this issue.  Sorry that things are not working for you.  For what it's worth, I conducted a mapathon on the 16th with iD 2.5.1 and about 50 people, all using a mix of browsers, and none of them experienced issues like this.
The loading spinner will spin as long as there are outstanding requests from the OSM API that have not been fulfilled.  The red api bar will appear when the OSM API server is unavailable or rate limited.
My best guess at the cause of these issues are a mix of a poor network connection, old Firefox and CentOS versions, maybe some broken usb drivers or dead mouse batteries could explain the issues with clicking either not working or clicks going to the wrong things.
If you have issues like this, iD might not be the best editor for your situation.  An editor like JOSM that works offline or over an intermittent network connection might work better.
Closing here, as I don't really know how to begin debugging this issue, but if you can add some error messages or specific details for me to try, please add here and I'll reopen..

Everything worked absolutely find until the 16th.  I used iD a lot prior to that.  No problems.
Firefox is the most recent available for CentOS 7.3.  It may not have the latest bling but security fixes are backported and its DOM-handling isn't significantly dated.  In any case, I also have this problem with Windows 10/Edge (and that was after the latest patch Tuesday updates).
Network connection is good enough to watch youtube without stuttering.  Usually at low-res because of data caps, but it works at high-res too.  It's not because of a slow connection.
USB drivers are fine.  The mice don't have batteries.  The clicks work fine to drag the map around.  The clicks work fine to operate the controls at the right of iD.  The clicks work fine with every other app on the computer and every other web page I look at.  The clicks are not able to select iD objects or operate the controls at the top even though they suffice to drag the map and operate the controls at the right.  This is not a mouse/USB problem.  This is definitely a "some part of the javascript isn't doing what it should" problem.
As I said, I found a work-around by bringing up the help walkthrough.  Once I exited that, everything about iD worked as it should.  So it's definitely not a connection/mouse/USB issue, because I've since made several successful edits.  Absolutely fine in all respects.  But it doesn't stay fixed.  Shut down and start back up and I have to go through the walkthrough tutorial again.  I've done that enough times to know that it consistently starts up borked and is consistently fixable with the walkthrough.  100% repeatable across two computers, two OSs, two browsers.
Which sorta implies some flag got prodded by the walkthrough.  Which makes me wonder about a race hazard somewhere.  Especially since you say you had a mapathon on the 16th without problems.  Especially as one other mapper tells me he's had the same problem a couple of times since the 15th, but it went away after he closed then re-opened his browser.  Especially as the computers I have at my disposal are rather old and not blindingly fast.  There's a lot of asynchronous stuff going on in the javascript, so my money is definitely on a race hazard.
My computer is tethered to my mobile for internet, so there can be occasional connectivity interruptions, which explains that API error, now you've told me what causes it.  A one-off and not relevant to the problems I have.  A red herring.  Forget I mentioned it.
I don't have a clue how to begin debugging it either. :)  But if you have any suggestions (that don't involve buying a new computer I can't afford), I'll give them a go.  Maybe if you can find a slower computer to play with...

I don't have a clue how to begin debugging it either. :) But if you have any suggestions (that don't involve buying a new computer I can't afford), I'll give them a go.

I would definitely appreciate any help you can lend - and it sounds like you are great at troubleshooting.
You can try:

Install iD locally or use the debuggable mirror at http://preview.ideditor.com/master/
set breakpoints in the code and see where the clicks are going / why the API calls in services/osm.js are failing

It does sound a lot like #4542 (maybe the same issue?)

Nope, I'm not great at troubleshooting.  For instance, I forgot to mention that I disabled Adblock Plus and uBlock Origin to no effect (first thing I tried).  I'm just about smart enough to understand when I'm on the wrong side of the Dunning-Kruger effect.
I tried the debuggable mirror.  Predictably, it worked.  Which means we're looking at a Heisenbug and almost certainly a timing problem is involved.
Also, I couldn't see what makes that, as opposed to the non-dev version, debuggable.  Unless the difference is squashed javascript as opposed to unsquashed.  If that's the difference, tracking down a timing bug would be very difficult.  Even more so because the bug isn't showing up in the debuggable mirror.
From what I can see, my symptoms aren't the same as #4542, but in case I misunderstood the other issue I've added a screencap of what I see when I hit the problem.  Both issues may have the same ultimate cause, though.
It's getting late here, and I still have things to do before bed.  So it will probably be tomorrow before I can look into it any further.  And maybe by then you'll have figured it out anyway and saved me the trouble. :)

After some playing and pondering, I can give you some additional info that may narrow it down a little.
This is all about the cursor (aka mouse pointer) and where it behaves differently.  I'm comparing standard 2.5.1 (borked) with 2.5.1 after the workaround of going to the walkthrough tutorial.
When the cursor is over a non-active part of the map, both show the grabby hand, indicating I can drag the map around (and I can do so).
When the cursor is over the controls at the right (layers, map data, zoom, etc.), in both it changes to a tappy finger and a tooltip appears.  The controls work.
When the cursor is over the controls at the top (point, line, area), in both it changes to a tappy finger and a tooltip appears.  The controls do nothing in the borked version but function correctly after the workaround.
Interesting is when I hover over an active part of the map (point, line, boundary of an area).  In the workaround version the pointer changes to a mini-arrow with a square box showing the type of object and the object itself is highlighted with a red outer boundary,   In the borked version, the pointer remains a grabby hand and no highlighting occurs.
Clicking on an object produces different results.  In the workaround version, the result is a flashing red boundary around the object and the object details appear in the left pane.  In the borked version no highlighted boundary (whether flashing or not) and no object details in the left pane.
Clicking and holding behaves the same as far as the cursor is concerned.  In both versions the "mini-arrow plus box containing object type indicator" turns into just a mini arrow.
That's about it, as far as I can tell.  But may narrow things down a little.
Further thoughts.  I initially suspected a race hazard and would have put money on the return of a JSON request before a handler had been set up for it.  But that is almost certainly wrong because the debugging version runs a lot slower on my computer but works fine.
You think this is similar to two other issues.  The one complaining about the spinner never stopping probably is.  I'm not so sure about the one where objects aren't displaying.  But there's a way to tell.  Ask them if my workaround fixes their problem. Even if they eventually turn out to have different causes, if the workaround fixes it then at least they can do some mapping.
Workaround is click on help.  Click on "sine-wave leading to X-marks-the-spot-on-the-folded-map" to get the walkthrough.  Leave the walkthrough.  iD is now fixed (until you close the browser window).

I'm glad you re-opened this, makes it a little easier for me to find. :)  The title change is an improvement in one way but not in another.  I get this exact same behaviour (including the workaround that fixes it) in Edge on Win 10.
I suspect the common factor is hardware age.  People tend to put Linux on older hardware that's no longer capable of running newer versions of Windows, so you're seeing this issue crop up on older hardware which just happens to be running Linux.
My main computer has a 1GHz Athlon 64 X2 Dual Core Processor 4000+. The laptop has Intel(R) Pentium(R) Dual  CPU  T3200  @ 2.00GHz, although /proc/cpuinfo tells me it's actually running at 1GHz.  They're not fast.  They have only two cores.  I doubt they've even heard of threading.  If there are race hazards in your code, this is the type of hardware it's going to show up on.

I'm experiencing the same problem and successfully using the same workaround. Was my first time mapping. Happens in both Firefox 57 and Chromium on Linux Mint XFCE 32bit 18.2. I'm on an Acer Aspire One netbook with an Intel Atom processor and screen resolution of 1024 x 600. Hope this helps.

I also have the problem that the spinny thing never stops spinning. Seen with 2.5.1 (but not with 2.5.0) on iOS 10 Safari (latest version on iPad) and Debian Linux 9.2, 32-bit i386 with Chromium and Firefox.

A couple more observations that might help narrow down where things are going wrong.
First, where it behaves correctly even when borked.  As stated before, hovering over an object does nothing (no red boundary, no info in the left pane), but I just found an exception.  If I hover over the label (street/road name) on a way, it obligingly hides itself just as it would on the working version.  So mouseover is bubbling up/down/sideways (depending on internals of the DOM) to the code.
Second, I was looking at a question in the forums.  The person had given the URL for the node he was talking about: [(https://www.openstreetmap.org/edit?node=4539676292#map=18/36.72868/-4.67321].  When I clicked on it, mirabile dictu everything worked.

Same problem after latest Firefox Update. I'm running Firefox 57 on Linux Xubuntu.

Same problem after latest Firefox Update. I'm running Firefox 57 on Linux Xubuntu.

Three questions.

Does the workaround fix it for you?  It's a pain but at least it lets you keep mapping.  It would also show if you're hitting the same problem as me or a different problem that happens to resemble it.

Are you able to downgrade Firefox to the version you had previously?  If so, does that fix it?  This is an important question if you're able to give it a try.

Is your computer old and/or low on memory?  I think that's a contributing factor, and why so few people are seeing this problem, but I could be wrong.

My experience is that this happened without a Firefox update; it happened after iD went to 2.5.1 on the 15th of the month.  I'm on Firefox 52.4.0 on CentOS 6.8 and on 7.3.  It also happened to me with Edge on Win10.  All computers at my disposal are low on memory and use older 64-bit processors.  It's happened to somebody with Safari/iOS and to somebody else with Chromium/Mint.  I think it's a race hazard introduced in iD 2.5.1, so I'd be surprised if it's also caused by upgrading to the latest Firefox (but life is full of surprises, especially when I try to figure out what's wrong with things).

Yes going through the walkthrough works.
Instead of downgrading Firefox, I tried it out on my faster laptop running exactly the same version of Firefox and Xubuntu and everything worked ok. I also tested it on Windows 8 and up to date Firefox and everything worked ok too.
The problem machine is an old EeePC 1015, Atom N550 1.5ghz Dual Core 2GB ram 64bit.

Hope this information helps!

@osmgit
Yes, it all helps to understand what's going on.  Thanks for responding.
Do you know when your Firefox upgrade happened?  You're not the only one to report a problem after a Firefox upgrade, but I think it almost certain it was a coincidence of timing.  Your answers make that close to certain, but a date for the upgrade would make it absolutely certain.
You also confirm it's happening on older machines with little RAM.  Which explains why only a few of us are seeing the problem and what a developer would need in order to be able to reproduce the problem.  I think it possible that even on a higher-spec box it may be possible to provoke the problem by having a lot of other apps running so there's a lot of paging slowing things down (but I could be wrong on that, it may be to do with numbers of cores and/or threading capability).
This is a nasty one to figure out, because the debugging version doesn't have the same problem.  It's a Heisenbug. :(

Firefox Quantum was released on November 14th I believe. So I probably updated sometime around then.
I think this might be related to the iD upgrade rather than the Firefox upgrade. I did some mapping at the start of this month and everything was working fine, then I got back into mapping a couple of days ago and this problem occurred.

Hey!  I just closed #4572, and I'm wondering if that might have also fixed this issue with Firefox on Linux or other platforms possibly susceptible to race conditions.
If you're experiencing this issue, can you test with the iD here:  http://preview.ideditor.com/master/  and let me know if the problems are still occurring?

I just tried it and it appears to be working fine.
However, isn't that the debuggable version?  You suggested I try that 2 weeks ago, before you made the changes, and it worked for me back then.  So maybe you've fixed the problem, maybe it's just that the debuggable version changes the timings so it works whether you've fixed the problem or not.
Do you have a non-debuggable version of that?

However, isn't that the debuggable version? You suggested I try that 2 weeks ago, before you made the changes, and it worked for me back then. So maybe you've fixed the problem, maybe it's just that the debuggable version changes the timings so it works whether you've fixed the problem or not.

Oh, sorry, I missed that above.  If the bug isn't reproducible in http://preview.ideditor.com/master, than I'm out of ideas.

This bug isn't reproducible in http://preview.ideditor.com/master but that doesn't mean you haven't fixed this bug, as well as #4572.  It just means that I can't tell if you've fixed it or not.  I won't know either way until you either formally release 2.5.2 or put 2.5.2 somewhere I can get at.
It's probably better to release 2.5.2 anyway.  Even if it doesn't fix this bug it will at least reduce the load on the tile servers.  So there's no point delaying things until you've found out how it affects this bug.

I've had the eternal spinner a couple times (Arch, Firefox 57). Button reload doesn't help, F5 has fixed it.

I am experiencing the same problem bhousel described.
My hardware is not That old, running 2.0 GHz i686 32-bit with 2Gb of RAM, but yes a single core. My browser has not changed from when it did work, and was installed in September.
My system is running: Linux i686 Debian 8.5 running PaleMoon 27.3.0SSE and the editing page loads in about 10 seconds. I switched from Firefox because this browser has less bloat and runs faster for nearly everything.
I also tried in Linux i686 Mint 17 (dual-boot) running Firefox 54.0 with same results.
I can only edit in Debian 8.5 running the newest Firefox ESR 52.5 but I have to wait 10 seconds between clicks as something in the javascript makes the cpu run at 100% load, and it no longer caches my clicks like it used to.
When fiddling with the mouse repeating the observed changes in the hand cursor, I stumbled on a workaround without loading help. Try this and see if it works for you, it might provide a clue :)
Click on the [-] box on the right side to zoom out, drag the map to pan in a direction a screen height/width. then click on the center [Press + to edit] box. Now click on a roadway and see if it lets you edit it.
I just followed the link to ID master, and after much zooming and panning to get home, it worked fine for me on the first road I clicked. Maybe I just repeated the workaround I found in doing so, since it didn't start zoomed in with edit already enabled.
Any other questions or ideas I could try for you?

@seahorse41
Your workaround also works for me.  In fact, I even simplified it.
Zoom out until you get the "Zoom in to edit" box then click on either of the pluses (the one at the right or the one in the box telling you to zoom in).  Fixed.  No dragging necessary (at least not for me).
Which probably explains the master.  You start zoomed out to the max and have to zoom in to do anything.
Which all ties in with bhousel's thought that maybe closing #4572 would fix this.  That problem was deferred tile loading where it was trying to get an unreasonable amount of tiles.  We can't be certain that's also behind this problem until 2.5.2 is released, but it's looking more and more likely.
I'm not sure about your slowdown, unless there's a lot of paging going on for some reason.  Takes a lot of open browser windows for my system to hit that problem, but then it hits a brick wall and occasionally craps itself.  But then I have 3G of RAM and you have only 2G.

Okay, sounds like the reported bug is getting handled.
As for the slow issue with newest firefox, should I open a new issue? I suspect something in the system libraries were updated this year causing widespread browser slowdowns, and so would not be a problem specific to this project. OSM Editing with older browsers is still snappy and loads fast.
My memory in use numbers are not even over 1Gb, so I doubt adding RAM would be relevant.
Thanks.

@seahorse41
I've not experienced a system slowdown, but I'm using an older release of CentOS (no systemd).  The iD editor broke for me when it went from 2.5.0 to 2.5.1 with no updates to anything on the computer in between.
I'm not clear, from what you wrote, if the browser that is experiencing slowness also needs the workaround to let you edit.  If it is, it may be the deferred tile fetches are causing the slowdown in the newer browser but not the older ones.  If it's a general slowdown with the newer browser then it probably isn't an iD issue.
To summarize: your guess is as good as mine. :)

@eehpcm
The browser experiencing the slowness, heavy cpu load, is Not affected by this bug, so does not need the workaround. It really is off topic here. Is there any way to debug what the firefox-57 browser is doing when it is spinning it's wheels in the IDeditor? What tiles would it be having trouble with?
I can tell that by disabling css, or js, or animations, that it clears up (speeds up) on other pages. I'm at a loss as to what the specific cause is, or who could help figure it out.

@seahorse41
The iD master is supposed to be debuggable.  But I never tried debugging because the issue didn't show up for me (I'd stumbled on your workaround without realizing it).
The tiles it's having trouble with are mentioned in issue  #4572.  If I understand it correctly, it is trying to fetch all the tiles.  All 315810 of them.  I didn't see network activity or paging that would indicate Firefox was actually fetching them, but I think it was going slightly bananas thinking about trying to fetch them.  Probably building a list of tiles to fetch or something like that.  With other setups it manages to complete whatever task is stalling on our machines and then realize that it doesn't actually need to prefetch 315810 tiles and can manage with actually fetching fewer.  Something like that.  Go read #4572 for more details.
Probably not a lot either of us can do except wait for 2.5.2 to come out and see if fixing the prefetch bug fixes what we're seeing too.  We have two workarounds.  Given the high zoom I usually work with I find my help tutorial workaround slightly faster than your zoom workaround.

For info, I'm seeing this too.  Things that may be relevant:

Seemingly slow network connection (but P2 can edit without problems)
Windows 7
Firefox 57.02
No enabled browser addons or plugins
Spinner going around all the time
Even when I edit at https://master.apis.dev.openstreetmap.org in an area where there is no data to fetch, the same problem occurs.


Does lucid work outside colab?

I am trying to run one of the ipynb outside colab in a juoyter notebook but visualizations are not showed, some model don't work.
I am using latest version of lucid I think is 0.3.8 and tensorflow-gpu 1.14.0

this may be related to the issue #229  i posted earlier, seems to be something related with the preprocessings of the input but i still cant say exactly what it is .

iam also using all libs in (seeminly) right versions, however the assertion error persists, the image_T_only produces Nans

yeah,it loses it's usability if you can only play in their environment

yes i tried to hack the whole thing out, even securing the dependency
version of numpy and tensoflow it does not works. Its get nans as outputs.
I think a path to uncover is see what make_T_vis and show functions are
doing. Seem a bit crazy to me that versions cant guarantee the workings,
but there may be more dependency in the backend we are not aware of ...
Please, if you can solve it feedback us, thank you.
On Fri, Feb 21, 2020 at 12:51 PM Cristina Segalin<EMAIL_ADDRESS>wrote:

yeah,it loses it's usability if you can only play in their environment
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/lucid/issues/230?email_source=notifications&email_token=AAOOCX47ZQZYGQNKHXXVZFLRD72B5A5CNFSM4KYZKUTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMTEYXQ#issuecomment-589712478,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOOCX32WUJGUK56JUY33BTRD72B5ANCNFSM4KYZKUTA
.

for me is more a matter of no being able to load some of the attributes in some modules or visualize correctly the objects and also assertions errors as well. If I had the time I would re-implement what I need but I am lacking of that at the moment.
Keep me posted as well, thank you

Which code are you trying to execute ? Are you able to go though the
tutorial, but outside Colab env ?
On Fri, Feb 21, 2020 at 1:07 PM Cristina Segalin<EMAIL_ADDRESS>wrote:

for me is more a matter of no being able to load some of the attributes in
some modules or visualize correctly the objects and also assertions errors
as well. If I had the time I would re-implement what I need but I am
lacking of that at the moment.
Keep me posted as well, thank you
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/lucid/issues/230?email_source=notifications&email_token=AAOOCX4BKOQ3DDJX2LRAC4LRD734VA5CNFSM4KYZKUTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMTGNIQ#issuecomment-589719202,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOOCX3DVCIJCZUHHA62HM3RD734VANCNFSM4KYZKUTA
.

I am playing around with the texture 3d synthesis. I can run in on colab but outside it's a mess.

I think this is the case for the other notebooks too. Basic tutorial goes
the same thing (and it is the basics ! its not as hard as 3D textures),
maybe there is a pre-processing taking place to adequate the code to
_img_url  function that helps to display in colab, but this preprocessing
is buggying others.
I ask the mainteiners and the code creator to try to revise it for
jupyter-notebooks in a plenty fresh installation, and see what happens ...
On Fri, Feb 21, 2020 at 1:18 PM Cristina Segalin<EMAIL_ADDRESS>wrote:

I am playing around with the texture 3d synthesis. I can run in on colab
but outside it's a mess.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/lucid/issues/230?email_source=notifications&email_token=AAOOCX657WMDGDO7C3NBZXDRD75EXA5CNFSM4KYZKUTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMTHRHQ#issuecomment-589723806,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOOCX337IXQBRUDIU4HI3TRD75EXANCNFSM4KYZKUTA
.

that would be really cool!!

I would start with the following strategy, try to execute in eager mode or
try to get rid of every tensorflow function that preprocess image, since
you have to start a session to get the tensor, this means the preprocessing
will take place only when you do the whole thing.
Them see what is going on with the number is numpy, once you solved the nan
problem, go back to tf functions.
But i still dont know if this gonna works, best regards !
On Fri, Feb 21, 2020 at 1:27 PM Cristina Segalin<EMAIL_ADDRESS>wrote:

that would be really cool!!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/lucid/issues/230?email_source=notifications&email_token=AAOOCX6LI4GTDOPHYES366DRD76ITA5CNFSM4KYZKUTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMTISPY#issuecomment-589728063,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOOCX26UVSSBQBX2DOUNBLRD76ITANCNFSM4KYZKUTA
.

try to put it before all the code you are executing
tf.enable_eager_execution()
tf.enable_resource_variables()
them go on hacking and seeing the variable values ...
On Fri, Feb 21, 2020 at 1:35 PM Daniel Penalva<EMAIL_ADDRESS>wrote:

I would start with the following strategy, try to execute in eager mode or
try to get rid of every tensorflow function that preprocess image, since
you have to start a session to get the tensor, this means the preprocessing
will take place only when you do the whole thing.
Them see what is going on with the number is numpy, once you solved the
nan problem, go back to tf functions.
But i still dont know if this gonna works, best regards !
On Fri, Feb 21, 2020 at 1:27 PM Cristina Segalin<EMAIL_ADDRESS>wrote:

that would be really cool!!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/lucid/issues/230?email_source=notifications&email_token=AAOOCX6LI4GTDOPHYES366DRD76ITA5CNFSM4KYZKUTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMTISPY#issuecomment-589728063,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOOCX26UVSSBQBX2DOUNBLRD76ITANCNFSM4KYZKUTA
.

or you can simply create a session with the graph in the context (the
supposition is it is alreaded builded by model keras instance), generate an
init_op and run it, then eval image, but seems an homeric turn around.
On Fri, Feb 21, 2020 at 1:46 PM Daniel Penalva<EMAIL_ADDRESS>wrote:

try to put it before all the code you are executing
tf.enable_eager_execution()
tf.enable_resource_variables()
them go on hacking and seeing the variable values ...
On Fri, Feb 21, 2020 at 1:35 PM Daniel Penalva<EMAIL_ADDRESS>wrote:

I would start with the following strategy, try to execute in eager mode
or try to get rid of every tensorflow function that preprocess image, since
you have to start a session to get the tensor, this means the preprocessing
will take place only when you do the whole thing.
Them see what is going on with the number is numpy, once you solved the
nan problem, go back to tf functions.
But i still dont know if this gonna works, best regards !
On Fri, Feb 21, 2020 at 1:27 PM Cristina Segalin <
<EMAIL_ADDRESS>wrote:

that would be really cool!!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/lucid/issues/230?email_source=notifications&email_token=AAOOCX6LI4GTDOPHYES366DRD76ITA5CNFSM4KYZKUTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMTISPY#issuecomment-589728063,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOOCX26UVSSBQBX2DOUNBLRD76ITANCNFSM4KYZKUTA
.

You can use something like this to try to debug the code while the devs are busy to not attend this bug.
def debug_info(tensor, mode="value"):
    import matplotlib.pyplot as plt
    with tf.Session(graph=tf.get_default_graph()) as sess:
        opinit=tf.global_variables_initializer()
        sess.run(opinit)
        t = tensor.eval()
        print(t.shape)
        print(t)
        plt.imshow((t[0,...]*255))

The thing iam wondering can be causing bug is not the pre-processing, since
it showed numerical matrices, but objective.as_objective can be breaking
the path of the layer or the tensor, returning something other than the
tensor it must be returned, but i cant look at this point right now.
C ya latter
On Fri, Feb 21, 2020 at 2:17 PM Daniel Penalva<EMAIL_ADDRESS>wrote:

or you can simply create a session with the graph in the context (the
supposition is it is alreaded builded by model keras instance), generate an
init_op and run it, then eval image, but seems an homeric turn around.
On Fri, Feb 21, 2020 at 1:46 PM Daniel Penalva<EMAIL_ADDRESS>wrote:

try to put it before all the code you are executing
tf.enable_eager_execution()
tf.enable_resource_variables()
them go on hacking and seeing the variable values ...
On Fri, Feb 21, 2020 at 1:35 PM Daniel Penalva<EMAIL_ADDRESS>wrote:

I would start with the following strategy, try to execute in eager mode
or try to get rid of every tensorflow function that preprocess image, since
you have to start a session to get the tensor, this means the preprocessing
will take place only when you do the whole thing.
Them see what is going on with the number is numpy, once you solved the
nan problem, go back to tf functions.
But i still dont know if this gonna works, best regards !
On Fri, Feb 21, 2020 at 1:27 PM Cristina Segalin <
<EMAIL_ADDRESS>wrote:

that would be really cool!!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/lucid/issues/230?email_source=notifications&email_token=AAOOCX6LI4GTDOPHYES366DRD76ITA5CNFSM4KYZKUTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMTISPY#issuecomment-589728063,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOOCX26UVSSBQBX2DOUNBLRD76ITANCNFSM4KYZKUTA
.

Thanks for reaching out @cristinasegalin and @Uiuran .
Lucid itself does not depend on colab, but code in the notebooks often relies on colab features to create visualizations of lucid's output. I personally use lucid outside of colab on a day to day basis, as do many of my colleagues.
In order to say more, I'd need to you to report specific failures with debugging details.
Please keep in mind that Lucid is research code. It's maintained by researchers actively engaged in interpretability research. We share it with the community in the hopes that it helpful, but we don't have the capacity to provide detailed user support or debugging.

Linus method. Thank you C. Olah.
One of the problems is that the Show function returns nan. However
make_t_image and transform_f evaluations returns numeric values, what make
me wonderer that the problem may be in objective.as_objective, that seems
to translate the name of the layer passed by in the function and return a
Tensor.
As I mentioned earlier, the first try in the function render_vis  of the
tutorial notebook return nans, the same happens to the notebook
of @cristinasegalin. This makes me think the problem may be the same.
You use in daily basis, but which parameter did you change to use in your
jupyter ?? (or you use in google colab version anyway ? unfortunately the
code is note open).
On Fri, Feb 21, 2020 at 4:43 PM Christopher Olah<EMAIL_ADDRESS>wrote:

Thanks for reaching out @cristinasegalin
https://github.com/cristinasegalin and @Uiuran
https://github.com/Uiuran .
Lucid itself does not depend on colab, but code in the notebooks often
relies on colab features to create visualizations of lucid's output. I
personally use lucid outside of colab on a day to day basis, as do many of
my colleagues.
In order to say more, I'd need to you to report specific failures with
debugging details.
Please keep in mind that Lucid is research code. It's maintained by
researchers actively engaged in interpretability research. We share it with
the community in the hopes that it helpful, but we don't have the capacity
to provide detailed user support or debugging.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/lucid/issues/230?email_source=notifications&email_token=AAOOCX5AVFTVR4BYKVV4UUDREAVEXA5CNFSM4KYZKUTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMT3RAI#issuecomment-589805697,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOOCX6MIPZBJCE4O2DPNZ3REAVEXANCNFSM4KYZKUTA
.

Or maybe in import_model since it depends on the name of the model and the
scope
On Fri, Feb 21, 2020 at 8:59 PM Daniel Penalva<EMAIL_ADDRESS>wrote:

Linus method. Thank you C. Olah.
One of the problems is that the Show function returns nan. However
make_t_image and transform_f evaluations returns numeric values, what make
me wonderer that the problem may be in objective.as_objective, that seems
to translate the name of the layer passed by in the function and return a
Tensor.
As I mentioned earlier, the first try in the function render_vis  of the
tutorial notebook return nans, the same happens to the notebook
of @cristinasegalin. This makes me think the problem may be the same.
You use in daily basis, but which parameter did you change to use in your
jupyter ?? (or you use in google colab version anyway ? unfortunately the
code is note open).
On Fri, Feb 21, 2020 at 4:43 PM Christopher Olah<EMAIL_ADDRESS>wrote:

Thanks for reaching out @cristinasegalin
https://github.com/cristinasegalin and @Uiuran
https://github.com/Uiuran .
Lucid itself does not depend on colab, but code in the notebooks often
relies on colab features to create visualizations of lucid's output. I
personally use lucid outside of colab on a day to day basis, as do many of
my colleagues.
In order to say more, I'd need to you to report specific failures with
debugging details.
Please keep in mind that Lucid is research code. It's maintained by
researchers actively engaged in interpretability research. We share it with
the community in the hopes that it helpful, but we don't have the capacity
to provide detailed user support or debugging.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/lucid/issues/230?email_source=notifications&email_token=AAOOCX5AVFTVR4BYKVV4UUDREAVEXA5CNFSM4KYZKUTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMT3RAI#issuecomment-589805697,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOOCX6MIPZBJCE4O2DPNZ3REAVEXANCNFSM4KYZKUTA
.

thank you again, iam trying the following (this is mentioned in the render_vis docstring)
with tf.get_default_graph().as_default() as graph, tf.Session() as sess:

   T = render.make_vis_T(model, "mixed4a:300")
   tf.initialize_all_variables().run()

   for i in range(500):
       if 10*(i//10)-i == 0:
         print(i)
       sess.run([T("vis_op").outputs[0]])
   print(T("input").eval()[0].shape)
   plt.imshow(255*T("input").eval()[0])

But something seens not to be working, since i only get gray images or nan (the algorithm is non deterministic in some place too ...)

i cant get the code to work, nor the same notebook of the tutorial, nor this simpler version (or more boring as stated in the docstring). As Colah said, there may be numerical stability issues, but it never happened to me when i was working on the deep dream code, so i will be back to that code, and as soon as this works outside colab, then i came back to contribute.
An aside: the mere fact that one thing works optimally in the cloud, but not outside of it, even with same dependency (apparently ...), is bizarre (of course that's the reason why we are researching). That's mean that there is something out there that is a dependency to Lucid.
Another way to see it is if the authors contrib with scripts that would work outside colab, as to exemplify what type of source code they thing does the job in any environment or .... add more dependencies that now are not visible.
I am really anxious to see this work, because doing Art work with the old original deep dream is pretty tedious, iam having to structure a lot of code in the place of editing-images time. Also, the old deep dream has a lot of gimmick magick such those random tiles and laplacians, slowly i wanna see it understabled and usable by digital artists. I know you guys are author of the old Deep Dream too, so i hope we can work on these together.
Regards.

Sorry for the late response I had to work on other stuff. I also took time to run all the colab notebook outside colab and they actually work except for some few parts where I think there are some discrepancies between python2 and python3 (so change few things to adjust) or using svelte as I haven't set that up.
So I wasn't fully able to run the building blocks and activation atlas notebooks because of that but all the other works actually outside on a jupyter notebook. I guess as there are some dependencies on ipython then some of the function may not work if just using regular python environment
the ones that for me don't work and I am interested in are actually the texture synthesis and style 3d.
When I run them and try to visualize the results nothing is shown. I think is a problem with opengl as I get an error when trying to define renderer = glrenderer.MeshRenderer((512, 512))  or trying to visualize show.textured_mesh

Can you post your configurations, versions and the versions of dependencies that you are running (i was not able to run nor texture 3D nor the basics). Maybe if i fix my dependencies, and maybe this would be placed as a commit to lucid install setup, than you can be helped with more hands.

Was there ever any resolution on this issue? I can't seem to get Lucid to run the code from the first tutorial outside of the collab
model = models.InceptionV1()
model.load_graphdef()
_ = render.render_vis(model, "mixed4a_pre_relu:476")
gives...
File "/home/emassey3/anaconda3/envs/tf1/lib/python3.7/site-packages/lucid/misc/io/serialize_array.py", line 51, in _normalize_array     assert not np.isnan(array).any() AssertionError