Automatic Export error (with Zotero Standalone)

I had an automatic export set for a collection, but it has failed : ID: 3RNZT5KX

I see only one error in the log:

Translator with ID 59cce211-9d77-4cdd-876d-6229ea20367f already loaded from "Bibliotheque et Archives Nationales du Quebec.js"

could it be you have a duplicate translator with that ID? BBT tries to reload the translators to install the bundled translators and it might be failing unceremoniously when it encounters this error.

Yes I've seen this error but it's old and unrelated to BBT I think.

But you're right this error is caused by a duplicate translator with the same ID (actually the same translator but with diacritics in its filename, something that Zotero has changed). I've deleted the litigious file. Still, auto export is not working, ID: R2V794XC.

At least now there's a different error. Is extended debugging enabled? Because I see nothing from BBT in the debug log. If it wasn't enabled, please enabled it and send a new debug log; if it was, it's an early startup error.

What about these two reports: WPZMH886 and DRRCF4KW ?
Yes, it's Standalone.
(also trying to create a new export for these tests, I've some encoding problem, but let's see that later)

In both WPZMH886 and DRRCF4KW I see

parent of candidate path<EMAIL_ADDRESS>exists but is not writable

This would mean that (despite what the message says) either the parent directory does not exist, or it is not writable.
I've checked in a change that distinguishes between these two error conditions. I'm a little worried about those unicode missing marks in that string -- the path is just what I've been handed the Zotero translator code. I do see elsewhere that BBT can access the correctly spelled folder (Grégoire and Thèse). But these paths will be in the autoexport config of BBT, so this will always fail (I think) unless these paths are fixed. The easiest way to do that is to delete the autoexport and capture it again.

Indeed, I can delete the autoexport and capture it again, but that's not really a solution :-)
What surprises me is that it used to work correctly before, even with diacritics in the path. And I've just tested it again with a slightly shorter path (still containing accented chars): "C:\Users\Grégoire\Desktop\test.bib", then with the same path and it seems to work (except the previous litigious autoexport)…!?
ID: 6RC44WGT ; 6HJM4TNV

6RC44WGT and 6HJM4TNV still have the same error report. Once the wrong paths are in the autoexport table there's no question on while they fail -- the question is how those broken paths got in there. In these reports, did you remove and re-add the auto-exports, or are these still the broken auto-exports you also had when you created  WPZMH886?

No I created these auto-exports for testing purpose, and immediately add/delete items from the collection to test the auto-export.

I've deleted all my auto-export, close Standalone, launch it again. If I create an auto-export, play with it, etc., it works well. The path is correct in the bbt prefs. But if if close and start Standalone again, the path in the prefs is changed with the accented chars replaced by the unicode missing marks…

Wot?
Can you go into the Zotero data directory (Preferences/Advanced/Files and Folders and click "Show Data Directory") and drop db.json file you find in the better-bibtex directory into https://dbinbox.com/retorquere?

Done. There's also db.json.1, db.json.2, db.json.3, db.json.4 and db.json.5 in the dir.

Those are backups, so that's OK. Could you try https://github.com/retorquere/zotero-better-bibtex/issues/537#issuecomment-233906169 again with this version?

Nothing in that log that seems relevant to BBT. There is an error in the log, but it seems unrelated to BBT -- please try this version which works around what appears to be a Zotero bug to see of the log now clears.
Could you get me a new copy of db.json on dbinbox, and a new error log with the version on the link?

Installing the first custom version (i.e. zotero-better-bibtex-1.6.68-circle-2618) deleted all db.json except db.json.5
ID, anyway: 4RS5EWQ3

4RS5EWQ3 has the same errors, so the workaround didn't do anything. That means that error must be before BBT initializes (I don't see the error, so I can't verify). Can you disable BBT, restart ZSA, and call up the error log from the Zotero advanced prefs to see if you see something like

[JavaScript Error: "TypeError: str.split is not a function" {file: "chrome://zotero/content/xpcom/utilities.js" line: 527}]

I have no idea how the db.json files disappeared. I'm going over my code but I just don't see a place that removes them -- specifically db.json.1-4
After restarting, and adding an auto-export, restarting again, there really must be a db.json file. Can you get me a copy of that?

I can't see the "str.split is not a function" error in the zotero debug output…

There's no db.json except db.json.5… I tried to reinstall the latest official version (1.6.68), create an auto-export etc, a db.json was recreated. But once I've updated to the latest custom version, it was removed.

Argh! I take it you do see them when bbt is enabled? Wait, did you have
debug logging on? In the zotero advanced prefs, tick "enable after
restart", restart, and see what's in the log then.

"enable after restart" was already checked in Z advanced prefs

Do you have an idea when it is removed? Before restart, after restart? I'm looking at the diff between the released version and the prereleases you've tried, and there's really nothing in those 5 lines that touches the backups. In the logs, I can see the backups being created.

It is created immediately after starting Zotero Standalone with 1.6.68 :  "db1.6.68.json" - 8.8 kB
It is modified after the installation of 1.6.68-circle-2619, but before restarting Zotero Standalone : "db-after_custom_install-before_restart.json" - 9.0 kB
It is deleted when Zotero Standalone is restarted after the installation (no more file to send you obviously but this ID: 5Q3XDMSW

I have a version lined up with logs to the console, but my build service (circleci) has an outage, I must wait for that to clear.

Alright, they're back up. The version can be found here. This version logs to the JS console, which should be available from the start vs the Zotero debugging which kicks in a little late for some things.
The console is usually limited for performance reasons, but since it is also a very busy place, messages often get lost before you can get at them. You can (temporarily, please change back later) the number of lines logged by going into about:config and upping the devtools.hud.loglimit.console limit. Change that to e.g. 20000, stop Zotero, start Zotero, and open the javascript console (you might have to fiddle with the command line to get that done), and filter on better-bibtex. I'm looking for any errors, but also specifically for lines that contain DBStore.

(in your latest log, the TypeError: str.split is not a function errors are gone BTW -- gotta love transient errors)

I'm afraid there's no devtools.hud.loglimit.console in Zotero Standalone about:config. I'll try with Z for firefox (I'm using both actually, and btw don't really understand the BBT warning "Zotero is in connector mode –– not saving database").

Nice. Does the console show anything relevant nonetheless?

With Z Firefox and circle-2625 :

1469050374496	addons.manager	WARN	Exception calling callback: TypeError: statusButton is null (chrome://zutilo/content/firefoxOverlay.js:327:9) JS Stack trace: getStatusPopup@firefoxOverlay.js:327:9 < setupStatusPopup@firefoxOverlay.js:334:25 < ZutiloChrome.firefoxOverlay.firefoxZoteroOverlay@firefoxOverlay.js:63:9 < ZutiloChrome.firefoxOverlay.init/invalid 'in' operand loadInfo
PolicyImplementation.asyncOnChannelRedirect()
contentPolicy.js:380
getContentsFromURL()
file.js:214
Zotero.BetterBibTeX.getContentsFromURL()
zotero-better-bibtex.js:1764
Zotero.BetterBibTeX.load()
zotero-better-bibtex.js:1778
Zotero.BetterBibTeX.loadTranslators()
zotero-better-bibtex.js:1468
Zotero.BetterBibTeX.init()
zotero-better-bibtex.js:991
load()
include.js:14
exports.Utils.yield()
utils.js:361
INIParser.prototype.process()
filterStorage.js:798
exports.IO.readFromFile/onProgress()
io.js:97
exports.IO.readFromFile/<()
io.js:182
InterpretGeneratorResume()
self-hosted:1040
next()
self-hosted:947
TaskImpl_run()
Task.jsm:319
bound TaskImpl_run()
self-hosted:792
Handler.prototype.process()
Promise-backend.js:937
this.PromiseWalker.walkerLoop()
Promise-backend.js:816
bound ()
self-hosted:750
bound bound ()
self-hosted:750
this.PromiseWalker.scheduleWalkerLoop/<()
Promise-backend.js:750
contentPolicy.js:380:0

Oh, eh, the latter warning means zotero is in connector mode. Zotero
triggers connector mode when both zotero Firefox and zotero standalone are
running simultaneously, in which case zotero Firefox behaves like the
connector plug-ins such as for chrome. This is mainly because having both
zoteros (and bbts) acting on the same database is likely to corrupt the
databases.

(I understand what is the connector mode ; what I don't get is the BBT behaviour in this case)

Same use-case as zotero itself - having both bbts acting in the same
profile is going to corrupt the bbt database.

(ok, but it seems to me that bbt displays this warning even if Z for Firefox is not started ; but I'm not sure and that's another issue).
What do you think of the report above : https://github.com/retorquere/zotero-better-bibtex/issues/537#issuecomment-234093652 ?

I get a notification from Zotero when it changes into connector mode -- when I get that, I follow suit (and back). So if BBT thinks it's in connector mode, it's because Zotero thinks it's in connector mode.
That's a strange error. Can you disable Zutilo temporarily and see if the problem goes away?

I start ZStandalone, disable Zutilo, restart, create an auto-export, close ZS, re-start it. The auto-export doesn't appear anymore in the BBT prefs. I close Standalone.
In Firefox, I open Zotero (Zutilo isn't disabled there) and get these two warnings (not once…):
"Citation key rescan after upgrade"
"DBStore : failed to load cache"
I'm using bbt circle 2625 in both Standalone and Firefox

Restarting Firefox with Zutilo disabled, the javascript console displays this (for better-bibtex filter):
invalid 'in' operand loadInfo
PolicyImplementation.asyncOnChannelRedirect()
contentPolicy.js:380
getContentsFromURL()
file.js:214
Zotero.BetterBibTeX.getContentsFromURL()
zotero-better-bibtex.js:1764
Zotero.BetterBibTeX.load()
zotero-better-bibtex.js:1778
Zotero.BetterBibTeX.loadTranslators()
zotero-better-bibtex.js:1468
Zotero.BetterBibTeX.init()
zotero-better-bibtex.js:991
load()
include.js:14
exports.Utils.yield()
utils.js:361
INIParser.prototype.process()
filterStorage.js:798
exports.IO.readFromFile/onProgress()
io.js:97
exports.IO.readFromFile/<()
io.js:182
InterpretGeneratorResume()
self-hosted:1040
next()
self-hosted:947
TaskImpl_run()
Task.jsm:319
bound TaskImpl_run()
self-hosted:792
Handler.prototype.process()
Promise-backend.js:937
this.PromiseWalker.walkerLoop()
Promise-backend.js:816
bound ()
self-hosted:750
bound bound ()
self-hosted:750
this.PromiseWalker.scheduleWalkerLoop/<()
Promise-backend.js:750
contentPolicy.js:380:0

BTW, you're on an older version of Zutilo (1.4.0pre001 where the current version is 2.0.2), and I think the error that was in the console log has since been fixed.
The "Citation scan" message is normal if you've installed a new BBT version, and each circle version counts as a new version.
I can't find the message "failed to load cache" in my code anywhere -- the closest I can find is DBStore: failed to load cache.json, but that would be followed by a reason for the failure. Was the log line perhaps wrapped?

The reason why I'm looking at Zutilo is that I've seen these kinds of things before -- other extensions running asynchronously (which is the in thing to do with javascript) concurrently (insofar that exists in javascript at all) with BBT calling Zotero.File.getContents, the async code erroring out, and interrupting the in-progress call to Zotero.File.getContents.
If the problem persists even without Zutilo, I'd need the full stack trace that the error above is embedded in.

and FWIW, in my zotero data directory\better-bibtex, I have two files: "cache.json.5" and db.json.5" and the "translator" directory. I mean it's not only db.json which has been removed but also cache.json… I hope that helps.

Yes it's probably "DBStore: failed to load cache.json", the message was partly hidden.

db.json and cache.json are handled analogously at startup, so what affects the one will affect the other. It's only after startup that their timing starts to diverge.
The message after DBStore: failed will give the reason of the failure. One failure could be that the file is missing, which means it's been deleted before the DB loader ever gets to it.
The translator directory is a remnant of older times and can be removed.

ok, what should I do then ?
The DBStore: failed message only appears when I switch between Firefox and Standalone.

ok, what should I do then ?
The DBStore: failed message only appears when I switch from Standalone to Firefox [edited].
In the jsconsole, I get this:
TypeError: Zotero.BetterBibTeX is undefined zotero-better-bibtex.js:1354:7
TypeError: Zotero.BetterBibTeX is undefined zotero-better-bibtex.js:1333:1

The undefined errors mean there's an early startup error preventing BBT from initializing. I'll need a real-time debug output log, but on Windows, that's hard work to do. Without it I don't really know what's going on alas.

Zotero Standalone, BBT circle-2625:
1st log: start standalone, create an auto-export, close standalone
2nd log: start standalone, create another auto-export (as it seems that the previous one has not been saved…), add new item to the auto-exported collection, then delete it from the collection (should trigger two auto-exports), close standalone.

I just found out why the reason as missing from the logs: I was showing it in a popup rather than logging it. New version that corrects that at https://github.com/retorquere/zotero-better-bibtex/releases/download/builds/zotero-better-bibtex-1.6.68-circle-2627.xpi.

So what do you want ? Real-time debug with this version ?

Yes please, for this procedure:

Start Zotero and verify db.json is present after start
Stop Zotero  and verify db.json is present after stop
Start zotero with a real-time debug log

Can you try https://github.com/retorquere/zotero-better-bibtex/releases/download/builds/zotero-better-bibtex-1.6.68-circle-2632.xpi ? I think this will fix the issue.

Sorry for the delay. Just tested circle-2632: it works perfectly well so far ! Well done and many many thanks ! (But there's no db.json.saving which appears in the better-bibtex directory)

It's renamed before you noticed it. If you look at the file system events it will have been there.

My tests are running again, I'll cut a new release when they're done (approx 20 minutes in the case of a release)

and super thanks for reporting this!

(and the help with debugging -- it was your suggestion that it was the backup mechanism that set me on the right path)

@gracile-fr do you still have Z4 operational? You have a pretty big library I think, can you check the file sizes for the BBT files in the data dir?

Yes. I need BBT so I'm stuck to Z4! There are 2259 items in My Library.
My BBT version is probably out of date (1.6.91).
In zotero data directory (no betterbibtex.sqlite, why?):

File
Size
Date

betterbibtex.sqlite.bak
1 933 312
29/10/2015 19:49

betterbibtex-lokijs.sqlite-journal
197 168
15/07/2016 19:53

betterbibtex-lokijs.sqlite.bak
196 608
15/07/2016 16:52

betterbibtex-lokijs.sqlite.failedmigration
196 608
15/07/2016 19:53

betterbibtex-lokijs.sqlite.1.bak
196 608
13/07/2016 19:39

betterbibtex-lokijs.sqlite-1.tmp
327 680
13/05/2016 17:19

betterbibtex-lokijs.sqlite-2.tmp
753 664
26/05/2016 14:59

In better-bibtex subdirectory:

File
Size
Date

cache.json
712 889
09/08/2017 16:19

cache.json.1
712 887
09/08/2017 16:00

cache.json.2
712 889
09/08/2017 02:05

cache.json.3
712 887
08/08/2017 17:55

cache.json.4
712 889
08/08/2017 17:40

cache.json.5
712 887
07/08/2017 16:11

db.json
184 267
09/08/2017 16:19

db.json.1
184 264
09/08/2017 15:59

db.json.2
184 267
09/08/2017 02:05

db.json.3
184 264
08/08/2017 17:54

db.json.4
184 264
08/08/2017 17:54

db.json.5
184 267
08/08/2017 17:40

A lot of those will be superfluous but I don't recall exactly in which version what went away; for 1.6.100 you only need the json files.
I had hoped you had more references (I recall someone having 20k but I don't know who it was); but an extrapolation would put 20k refs at 7MB for the cache, and 2MB for the database. That's not too bad. I'm contemplating whether I should start saving data in the main DB for BBT5; the main DB takes care of backups, so I wouldn't have to do it myself, but it's async so I need to make sure the DB is ready before anything else happens (which is not as easy as it should be)
Anyhow, thanks, it's still a data point.

@retorquere, I have a portable installation of Zotero 4, so if you need data from a bit bigger library (3101 references), here they are:
 Directory of c:\Users\deni\zoteroPortable

2015-06-15  12:50                53 better-bibtex-serialized-items.json
2016-07-11  14:34           754˙360 betterbibtex-lokijs.sqlite-journal
2016-06-24  12:17        13˙664˙256 betterbibtex-lokijs.sqlite.1.bak
2016-07-11  13:02        13˙664˙256 betterbibtex-lokijs.sqlite.bak
2015-03-04  15:59           491˙520 betterbibtex.sqlite.1.bak
2015-10-29  20:47           786˙432 betterbibtex.sqlite.bak

 Directory of c:\Users\deni\zoteroPortable\better-bibtex

2017-08-09  16:54        14˙567˙013 cache.json
2017-08-08  20:24        14˙567˙014 cache.json.1
2017-08-08  20:23        14˙567˙012 cache.json.2
2017-08-07  15:48        14˙614˙384 cache.json.3
2017-08-07  12:54        14˙620˙856 cache.json.4
2017-08-07  12:54        14˙605˙326 cache.json.5
2017-08-09  16:54           643˙955 db.json
2017-08-09  16:53           643˙953 db.json.1
2017-08-08  20:24           643˙955 db.json.2
2017-08-08  20:23           643˙954 db.json.3
2017-08-08  20:22           643˙947 db.json.4
2017-08-07  15:48           636˙471 db.json.5
              12 File(s)     91˙397˙840 bytes
               2 Dir(s)  500˙791˙230˙464 bytes free

Is that a 14mb cache from 3.5k references? Looking only at the .json files.

Yes, I don't know why. After resetting the cache, cache.json.1 is 2,3mb, cache.json still 14mb. Entire library, exported do Biblatex (with abstracts) is about 3,4mb.

I'd have to look into that. Still, those aren't numbers I'd want to do deep voodoo for just to shave a little time off saving.


add histogram back

Motivation
Forgot to add the headways histogram back.
Changes
Adds headways histogram back.
Removing the grid in #748 which makes the whitespace at the botom go away.

Testing Instructions

I don't think that is realistic. To do that we need all the charts be the exact same height, which is the issue with the grid.

I don't love using "histogram" in the visible display text. Maybe "distribution"?


PM2-gui in remote server

Hello, I'm trying your software and its a very nice open source alternative.
I have tested it in a VPS, and I have it running wihtout problem. However, I want to know if there is a way PM2-gui is running a machine, and monitoring another machine(s).
Is there a way this can be done with this software? I suspect the remotes section is for that, but I don't have clue on how to use it.
Thanks!

it's definitely YES

run pm2-gui agent on your agent machines
serve a pm2-gui portal on your web machine by pm2-gui start
make sure the connections of agents can be resolved everywhere (<ws|http>://:)
just visit <your web ip address>:<port>...


 Thank you.
Can you please also mark the to-do as done in Readme as part of this PR?

Thanks for your reply, I marked the to-do as done in Readme now.

Input:
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
test '';
}

Before:
map $http_upgrade $connection_upgrade {
	default upgrade;
	close;
	test '';
}

After:
map $http_upgrade $connection_upgrade {
	default upgrade;
	'' close;
	test '';
}


Fail to start in Windows 10.

I installed go, latest version and used the command for windows go get -u github.com/miguelmota/cointop. It did not show any error and came to next line.
I used cointop to start but error shows up.
C:\Users\RohitSingh>go get -u github.com/miguelmota/cointop

C:\Users\RohitSingh>cointop
time="2019-06-11T21:11:07+05:30" level=fatal msg="mkdir : The system cannot find the path specified."

@Rohit036 Thanks for reporting the bug. Will debug when I get access to a Windows machine

this bug also appears when cointop.exe is run in Windows 7

@Rohit036 @rcdosado
There's new updates regarding the mkdir command for creating the config file.
Can you try go get -u github.com/miguelmota/cointop again? Thanks

Alright finally tested it on a Windows machine. Should be good now. Thank you for your patience!

Had to do
mkdir c:\tmp


fix: updated js-gardener

:tada: This PR is included in version 1.2.4 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Navigation keys are not working in confirmation modals

Description
When you try to delete something (table, schema, database, connection) you are not able to confirm the operation with the keyboard. For some reason right and left keys are not working and you are forced to use the mouse.

DBeaver Version
Community Edition 24.1.0
Operating System
openSUSE Tumbleweed
Database and driver
This issue is not related with the database or driver.
Steps to reproduce

Create a new database
Right click and try to delete the database.
Press left and right keys in the keyboard to try to confirm the action.

Additional context
No response

Thank you for the report. I experience the same issue on macOS, but have not found the time to file a ticket.

I experience the same issue on macOS, but have not found the time to file a ticket.

On Mac you have to turn on the "Keyboard navigation” setting to use the tab key.
See https://support.apple.com/en-gb/guide/mac-help/mchlp1399/mac

I'm not sure if this is a bug. You can cycle through buttons with tab on keyboard, but can't use arrow keys like it's possible on windows. But that seems to work as expected for Linux

Conclusion:

On Linux, you can use the tab button as a navigation key in confirmation modals.
On macOS, you can turn on "Keyboard navigation” and then the tab button will also work.
It seems that everything works as expected, so I closed this ticket.

Navigation is possible on macOS. However, hitting enter on the 'Yes' button doesn't produce the desired effect.
https://github.com/dbeaver/dbeaver/assets/26031301/33bcccfe-83e0-4a92-92f5-aab85990bba0

Navigation is possible on macOS. However, hitting enter on the 'Yes' button doesn't produce the desired effect.

That's because the Cancel button is set as the default button. To select "Yes" use the space bar when it has the focus.

Thanks a lot, @Phillipus


execute call

Im getting this error
hope you can help....
thanks
async _executeCall(api, params, body) {
^^^^^^^^^^^^
SyntaxError: Unexpected identifier
at exports.runInThisContext (vm.js:53:16)
at Module._compile (module.js:374:25)
at Object.Module._extensions..js (module.js:417:10)
at Module.load (module.js:344:32)
at Function.Module._load (module.js:301:12)
at Module.require (module.js:354:17)
at require (internal/module.js:12:17)
at Object. (/home/knuvihs/wired/_develop/homeconnect/home-connect.js:3:13)
at Module._compile (module.js:410:26)
at Object.Module._extensions..js (module.js:417:10)

Have a look at #1. You’ll need a recent version of NodeJS for async/await.

thanks!!
i got further now
but now i have this, looks like a auth problem on homeconnect
but expecting not an error...
https://developer.home-connect.com/security/oauth/authorize?client_id=2239DA12410650871D1836B36DDA734736A9A55D7A3A28B1B20E505692F082&redirect_uri=https://apiclient.home-connect.com/o2c.html&response_type=code&scope=IdentifyAppliance Monitor *-Control Settings&state=250da41b-f9ea-400c-ba17-68ab50ddb74c
Error: Invalid response 403
at Request.request.get.on.on (/home/knu/wired/homeconnect/index.js:103:20)
at Request.emit (events.js:160:13)
at Request.onRequestResponse (/home/knu/node_modules/request/request.js:1068:10)
at ClientRequest.emit (events.js:160:13)
at HTTPParser.parserOnIncomingClient (_http_client.js:548:21)
at HTTPParser.parserOnHeadersComplete (_http_common.js:115:23)
at TLSSocket.socketOnData (_http_client.js:439:20)
at TLSSocket.emit (events.js:160:13)
at addChunk (_stream_readable.js:269:12)
at readableAddChunk (_stream_readable.js:256:11)

ahhh advanced account i think
but need to pay for that... right...

Simulator API endpoint was changed. Updated the code, thanks.

Ok working thanks:-)
working on on the simulator.
Final question: can you help me pointing out the real api settings
do i need to change these?
this.baseUrl = url || 'https://simulator.home-connect.com'
this.redirect = redirect || 'https://apiclient.home-connect.com/o2c.html'
I tried setting it to https://api.home-connect.com
Thanks!

Sorry, forgot to sync the code change - done now.
You don't need to update the code. The url is a parameter when creating the class, so set it there.
"API of physical home appliances (access limited to select partners - stay tuned for the public release)"

Looks like they are opening it up, physical device access: 'https://api.home-connect.com/api/'

I thought the real API was open yet, but still not I guess, I send them a mail to ask when it opens up.
taking to long now :-(
all is working again on the simulator
thanks for the great code!


build: upgrade to C++17

Signed-off-by: Lizan Zhou<EMAIL_ADDRESS>Description:
Risk Level: Med
Testing: CI
Docs Changes: N/A
Release Notes:
Fixes #9091

Hey, let me know how best to patch CEL OSS build. copts is added via copybara, so I can change it easily.

Have we confirmed this will work for Envoy Mobile? cc @keith @goaway @junr03. I think not having filesystem is fine on that platform, but we do need to not run into potential runtime issues if for example the C++ stdlib is dynamically linked on that platform (I'm not sure if it is or it isn't).

That's why this is draft, I haven't even confirm on all CI yet.
Based on my research iOS and Android should be fine with C++17 (without filesystem on iOS yet), but yeah need @keith @junr03 @goaway to confirm.

@kyessenov Thanks, yeah I'm not intended to merge with the patch and will sync with you for that.

We have not verified yet. I can work with @goaway and @keith to do so for both platforms. @lizan what is your timeline for upgrading? Just so I know how to prioritize this.

Definitely worth a try with enovy-mobile

@junr03 I'm out next week, so the week after?

For posterity: a couple weeks ago I tried @lizan's suggestion above and compiled Envoy Mobile successfully for iOS. However, there were a couple runtime issues (which in honesty I did not dig into due to lack of time).
Instead, @lizan, @goaway, and I had a conversation about prioritization and timeline for getting C++17 support into Envoy and hence Envoy Mobile. The team at Lyft (Envoy Mobile maintainers) is working steadfast for a production release of Envoy Mobile, and thus has few resources to spare. Given that there is not an urgent forcing function to upgrade @lizan agreed that we could deprioritize bringing C++17 support for several weeks. The Envoy Mobile team would have more spare cycles to resolved remaining unknowns sometime in late march.
If anyone reading this PR has any concerns or needs that are addressed by a faster upgrade timeline please drop a comment here and let us know what your needs are.

@junr03 @goaway any progress on Envoy mobile side?

Unfortunately, no. We have not had the chance to prioritize this. Any changes in timeline for you, i.e., should we prioritize this?

This is a blocker of https://github.com/envoyproxy/envoy/issues/10053, but we can explore to make Envoy build under both C++14/17 with compile_time_options.

I'd just like to /bump this issue.
@sunjayBhatia and I are looking to purse the FilesystemImpl in Envoy, ensuring we can grok the error results in a consistent manner, and that we can have utf-8 file-system naming conventions as on Linux. But it seems foolish to attack our pet project without first addressing whether we can get to a uniform std::filesystem ahead of those design decisions.
Can you enumerate the current blockers, @lizan ?

@stedsome is working on it.


Reading/writing text files as string

Those two classic functions I copy-pasted who knows how many times (except with updated buffer check):
/// @function file_read_all_text(filename)
/// @description Reads entire content of a given file as a string, or returns undefined if the file doesn't exist.
/// @param {String} filename        The path of the file to read the content of.
function file_read_all_text(_filename) {
    var _buffer = buffer_load(_filename);
    if (buffer_exists(_buffer))
        return undefined;

    var _result = buffer_read(_buffer, buffer_string);
    buffer_delete(_buffer);
    return _result;
}

/// @function file_write_all_text(filename,content)
/// @description Creates or overwrites a given file with the given string content.
/// @param {String} filename        The path of the file to create/overwrite.
/// @param {String} content         The content to create/overwrite the file with.
function file_write_all_text(_filename, _content) {
    var _buffer = buffer_create(string_length(_content), buffer_grow, 1);
    buffer_write(_buffer, buffer_string, _content);
    buffer_save(_buffer, _filename);
    buffer_delete(_buffer);
}

Naming comes from C# File.ReadAllText and File.WriteAllText, but I'm open to other name suggestions.
Technically, file_text_read_all and file_text_write_all would be more in line with file_text_* functions, but I wonder if it wouldn't be misleading, considering that most file_text_* functions require the text file handle, while the proposed functions are completely independent from file_text_* way of doing things. :thinking:
Could be used for JSON saving/loading as well, vide: https://github.com/Alphish/gm-community-toolbox/issues/9

If you go ahead with this, I would suggest using exception handling to protect buffer_load instead of file_exists. Aside from the file not existing, there are other things such as read permissions or conflicting access that can prevent buffer_load from working.
/// @function file_read_all_text(filename)
/// @description Reads entire content of a given file as a string, or returns undefined if the file doesn't exist.
/// @param {String} filename The path of the file to read the content of.
function file_read_all_text(_filename) {
    var _buffer;
    try {
        _buffer = buffer_load(_filename);
    } catch (ex) {
        return undefined;
    }
    var _result = buffer_read(_buffer, buffer_string);
    buffer_delete(_buffer);
    return _result;
}

While this feature didn't gather many upvotes, issue #9 has gathered quite a few, and reading/writing files to entire strings is required for JSON saving/loading.
Thus, I decided to put up this feature for name voting of sorts. There weren't ideas for other names, but maybe someone will come up with some:

🚀 for file_read_all_text() and file_write_all_text() (names inspired by equivalent C# functions, as mentioned above)
👀 for something else (please write what other name suggestion you have)
😕 for not implementing this function at all, and instead embedding the file string reading/writing in the JSON saving/loading functions

The feature has been accepted for the Community Toolbox.
file_read_all_text(filename) should be implemented so that it returns string contents of the file if available, or undefined if the load fails for whatever reason. It should be noted that buffer_load will return -1 if the file cannot be loaded, so it should be handled by buffer_exists check rather than try/catch block; I updated the first post accordingly. file_write_all_text(filename,content) should be implemented so that it saves a file with the given string. We will leave detecting and handling potential save failures to the user, as buffer_save doesn't seem to provide mechanisms for that out of the box.
file_read_all_text(...) tests need to check if the strings are read correctly from some Included Files. I would rather not make tests for file_write_all_text(...), because there's some risk of accidentally making files that would be left over (even if the code is written to delete the unneeded files, something like power outage might interrupt the test or something, leading to messy state).
The demo should include:

some text area with a string displayed
a button to change the content of the text area
a button to replace text area contents with a string read from a user-selected file (via some get open filename async function)
a button to save text area contents into a user-selected file (via some get save filename async function)

As usual, if someone wants to pick it up for development, please mention it here so that multiple people don't start the same thing independently.

I'm taking this one, hope to get it done soon.

Merged.


Use as much context as possible without exceeding max context window

Attempts to address the 2nd half of Issue #133

Go in reverse order to get previous instructions and responses
Continue building the full prompt until we exceed the context window size
Use the version of the prompt that does not exceed the context window size

@nsarrazin I'm not sure exactly how to calculate the token size. I just used what I understood, but if there's a helper method or better way to understand token size please let me know. This worked in my testing. There might be a more efficient or cleaner way of doing it, but I think the logic should hold.

As a result of this change I was able to continue a long conversation without encountering the same bug as in #133

As a result of this change I was able to continue a long conversation without encountering the same bug as in #133

Hi
Is this going to be merged? I constantly need to delete my chats to be able to add prompts more than once in a chat.

I tried this PR but I'm still getting the same issue and some of my questions appear and disappear in a weird fashion :/
I will try to refactor the backend of serge to use the python bindings  since they expose a tokenize method we could use that to get a definitive answer on how many tokens we have and cut appropriately.

I also tried this PR and encountered the same issue (albeit maybe less frequently.)

@nsarrazin Ah, I see. This is likely due to guessing the number of tokens instead of getting the actual number of tokens.
Does your other PR #143 supersede this change?
Thanks for confirming @kindrowboat


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-archive-test-source-version-target/pull/248 is merged into live branch.
Auto archive content to https://github.com/opstest2/docs-archive-test-source-version-target.git

Docs Build status updates of commit 9b30b02:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

.openpublishing.redirection.json
:white_check_mark:Succeeded

versioningArchiveSourceFolder/TOC.yml
:white_check_mark:Succeeded
View (sql-server-2014 || azuresqldb-current || azuresqldb-mi-current || azure-sqldw-latest || >= aps-pdw-2016 || >= sql-server-2016 || >= sql-server-linux-2017 || sql-server-previousversions || sqlallproducts-allversions)

versioningArchiveSourceFolder/docfx.json
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Running OPT 175B with different hardware configurations

❓ Questions and Help
Before asking:

search the issues.
search the docs.

What is your question?
How can I get the 175B model running for inference on a hardware setup as described below? Is it possible on one node with 8 A6000s with 51GB each, perhaps with DeepSpeed or similar? I know there are multiple other similar issues, but I'm wondering if the requirements can be somewhat relaxed for inference only (and my hardware setup is a bit different), so I thought I'd throw my question into the ring :).
What's your environment?

metaseq Version (e.g., 1.0 or master): master
PyTorch Version (e.g., 1.0) 1.10.1+cu113
OS (e.g., Linux): Linux
How you installed metaseq (pip, source): per instructions in https://github.com/facebookresearch/metaseq/blob/main/docs/setup.md
Build command you used (if compiling from source):
Python version: 3.9
CUDA/cuDNN version: 11.3
GPU models and configuration: (potentially 2 nodes of) 8x NVIDIA RTX A6000 51GB RAM
Any other relevant information:

delet

I have not personally tried, but I believe you might be able to do it for inference since the idling VRAM usage is <50GB as far as I remember. I don't think you need to make any changes to constants.py since it's 8 devices by default.

@sachit-menon @xhluca
Hi, We recently integrate OPT-175B with the Berkeley-backed system Alpa.
I guess you can try Alpa which exactly allows you to use more heterogeneous GPU setups to train/serve big models like OPT-175B, other than requiring 8x 80GB A100.
See this guide for more details!

@patrickvonplaten sorry if this is obvious, but where can I find those scripts? Also great to hear about accelerate, do you know where we can find example code?
I tried running the official tutorial and it seems that only a single gpu was used, so I'm not sure if it's because I did something wrong or there some special steps needed for those large models.

Hey @xhluca,
Sorry to answer so late - we're working with the meta team to make the HF weights directly available with a short code snippet on how to run them :-)

I'll keep you updated here

Thanks!

We've also got model parallel 16 working now, which is useful for running across two nodes (with some perf penalty)

Just as a side-note, we've converted the 175B checkpoint internally and it runs very well with accelerate on 8 A100s (faster or as fast as deepspeed for single-input queries).

8 A100s with 40GB or 80GB memory?

@samuelstevens Based on personal experiments, it takes ~50GB of VRAM per device, so likely done on A100 80GB.

@patrickvonplaten @stephenroller any idea what kind of timeframe that might happen in? How involved is it to use the conversion script/what are the steps involved in that, if it would be faster to do it that way? Thanks for your efforts on this!

In metaseq we now have support for MP16 which lets it work on 16x 32gb nodes. @punitkoura also has a PR for converting the weights for use with HF accelerate which supports much smaller machines via offloading (at a speed cost)

@punitkoura would it be possible to have a brief example/explanation of how to use that with the OPT 175B model? Digging into these commits, I found https://github.com/facebookresearch/metaseq/blob/main/gpu_tests/test_hf_compatibility.py with  associated test setup download_and_configure_125m_with_hf_dependencies in https://github.com/facebookresearch/metaseq/blob/main/.circleci/config.yml. However, I'm not sure if 175B has the dependencies or config files that 125M has available, so I don't know if following the exact same steps will work as-is.
Perhaps @patrickvonplaten has used this updated conversion script to put together that code snippet? 😄

Hey @xhluca,
Sorry to answer so late - we're working with the meta team to make the HF weights directly available with a short code snippet on how to run them :-)

@patrickvonplaten Since it might take some time to make it available, in the mean time is it possible to have a code snippet for the publicly available large models (30B and 60B)? This way, when 175B will be made available, the only thing needed is to change a single line of code.

@patrickvonplaten Bump on @xhluca 's comment above ? Sorry to ping you again, it's been about a month and I have some experiments that I'd really like to run!

Hey!
The 30b and 60b models can very easily be run if you have enough GPUs available. You can just follow the code snippets here:
https://huggingface.co/facebook/opt-30b#how-to-use
and
https://huggingface.co/facebook/opt-60b#how-to-use

Note that this assumes you have a 80GB A100 available. If instead you have just multiple GPUs available you can replace:
model = AutoModelForCausalLM.from_pretrained("facebook/opt-66b", torch_dtype=torch.float16).cuda()

by:
model = AutoModelForCausalLM.from_pretrained("facebook/opt-66b", torch_dtype=torch.float16, device_map="auto")

which will automatically place layers on the different devices in a smart way. Also see: https://huggingface.co/docs/transformers/v4.21.2/en/main_classes/model#large-model-loading

Is an easy integration with 175B still planned or shelved for now?

@stephenroller - could we provide you with the HF 175B checkpoint and you provide it somehow on your website (upon request)?

Yes Patrick, that would significantly help unblock this.

Awesome - send you an email :-)


Update-OctopusVariableSet crashing when adding environment

When running Update-OctopusVariableSet with an environment specified in the json-file the following error occurs:
Unable to find type [Octopus.platform.Model.Scopefield].
At C:\Program Files\WindowsPowerShell\Modules\octoposh\0.4.6\scripts\Update-OctopusVariableSet.ps1:97 char:25
+ ...             $newvar.Scope.Add([Octopus.platform.Model.Scopefield]::$s ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (Octopus.platform.Model.Scopefield:TypeName) [], RuntimeException
    + FullyQualifiedErrorId : TypeNotFound

[Octopus.platform.Model.Scopefield] should perhaps be [Octopus.Client.Model.ScopeField]?

This was fixed in 0.5.1 thanks to a PR from someone in the community: https://github.com/Dalmirog/OctoPosh/commit/780eddee25a12389453e80da47ee863c78a39b45
Can you please try in that version?

Closing Issue but willing to open it again if the issue persists.
You can also contact me @ https://gitter.im/Dalmirog/OctoPosh for questions.


EA Resources: fix links before hard launch

Currently:

the links on EA Resources to the hub direct to https://eahub.azurewebsites.net rather than eahub.org.
the link on the hub to EA Resources direct to https://confident-minsky-f713d6.netlify.com/, rather than resources.eahub.org

Fix this before we go live properly

@Manoj-nathwani @taymonbeal
Which pages are we launching with on Monday (18/2)?

groups.eahub.org
staging.eahub.org
both?

Need to know so that I can update the links from resources.eahub.org to the hub accordingly.
Also, should I remove any links to /profile on resources.eahub.org as we're launching without profiles? Or is that not necessary?

eahub.org should be the domain we're using for going live with the hub
we also have groups.eahub.org which @taymonbeal sorted out as a last minute to allow another org to access our data, but they're not a priority.

Don't link anything to groups.eahub.org, that's just going to redirect to eahub.org after the launch.

now linking to eahub.org. This is ready to go live at the push of a button

I'd just like to add that currently only navmenu resources link links to https://confident-minsky-f713d6.netlify.com other links link to https://resources.eahub.org

Updated navmenu, thanks @piotr-asdf
(this is on master, but not deployed. Will be visible with the next deployment)

@sebgrebe should we close this by now?

still haven't made the "new" resources live - this will be done once eahub goes live and can be done by accepting this PR. Don't want to close this issue so that we don't forget about it :)

Done: resources.eahub.org


(maint) Pin ffi to < 1.14.0

This updates the gemspec to pin ffi to < 1.14.0, which includes some
breaking changes on Windows. Previously, ffi was pinned to ~> 1.13.0,
which caused our packaging pipeline to fail, as bolt-runtime only
includes ffi 1.9.25.
!no-release-note

Just a note that pinning ffi to < 1.14.0 will block future Apple Silicon support. 1.14.0 was the first release to include support for the new architecture (see: https://github.com/ffi/ffi/issues/841.)

Just a note that pinning ffi to < 1.14.0 will block future Apple Silicon support. 1.14.0 was the first release to include support for the new architecture (see: https://github.com/ffi/ffi/issues/841.)

Oh, that is great to know! I think we'll get lucky with this one, as we can unpin this once we release Bolt 3.0 with Ruby 2.7 minimum version, which should be before we're able to support Big Sur or Silicon (as we're waiting on the infrastructure for testing the platform and hardware). But if either of those doesn't happen it's great to be aware of!

Oh, that is great to know! I think we'll get lucky with this one, as we can unpin this once we release Bolt 3.0 with Ruby 2.7 minimum version, which should be before we're able to support Big Sur or Silicon (as we're waiting on the infrastructure for testing the platform and hardware). But if either of those doesn't happen it's great to be aware of!


reduce precision to float32?

the benchmark is in float32, the forecasts are in float64.
I did a rough check https://github.com/judithberner/climpred_CESM1_S2S/issues/4 and https://renkulab.io/gitlab/aaron.spring/s2s-ai-competition-bootstrap/-/blob/master/notebooks/verification_RPSS-precision.ipynb.
I think we can massively save computing time and cloud storage if we go to float32.
for v in varlist_obs:
    for d in [dates[0]]:
        for dsl in obs_dataset_labels:
            # only netcdf, no choice
            try:
                if 'input' in dsl:
                    ds = cml.load_dataset(f"s2s-ai-challenge-{dsl}", origin='ecmwf', date=d, parameter=v).to_xarray()
                elif 'bench' in dsl:
                    ds = cml.load_dataset(f"s2s-ai-challenge-{dsl}", parameter=v, weeks='34').to_xarray()
                else:
                    ds = cml.load_dataset(f"s2s-ai-challenge-{dsl}", date=d, parameter=v).to_xarray()
                for v in ds.data_vars:
                    print(dsl,v,ds[v].dtype)
            except Exception as e:
                print(dsl,v, type(e).__name__,e)

training-output-reference tp float64
test-output-reference tp float64
training-input tp FileNotFoundError [Errno 2] No such file or directory: '/opt/conda/lib/python3.8/site-packages/climetlab_s2s_ai_challenge/training_input.yaml'
test-input tp FileNotFoundError [Errno 2] No such file or directory: '/opt/conda/lib/python3.8/site-packages/climetlab_s2s_ai_challenge/test_input.yaml'
test-output-benchmark tp float32
training-output-reference t2m float64
test-output-reference t2m float64
training-input t2m FileNotFoundError [Errno 2] No such file or directory: '/opt/conda/lib/python3.8/site-packages/climetlab_s2s_ai_challenge/training_input.yaml'
test-input t2m FileNotFoundError [Errno 2] No such file or directory: '/opt/conda/lib/python3.8/site-packages/climetlab_s2s_ai_challenge/test_input.yaml'
test-output-benchmark t2m float32

ds = cml.load_dataset(f"s2s-ai-challenge-training-output-reference", date=d, parameter=v).to_xarray()
print(ds[v].nbytes, ds[v].astype('float32').nbytes)
213734400 106867200

Thoughts @floriankrb @b8raoult ?

hi @floriankrb, see the gist here: https://gist.github.com/aaronspring/5182caf232a5d103b07322cf0bdfd191 Is this the test/demo you asked for? I would be just about the fore/hindcast-like-observations. ds=ds.astype('float32') is all we need in the script IMO. Am I missing something?


Add a disabled rustfmt.toml

We do not currently use rustfmt, this is a nuisance for devs who routinely work on code bases that do use rustfmt who often have their editors set to format on save. We can make the life of such devs much better by explicitly disabling formatting using rustfmt.toml.
ref: https://rust-lang.github.io/rustfmt/?version=v1.4.38&search=#disable_all_formatting

Ran cargo fmt locally and it did not do anything.

Perhaps it makes sense to also add a CI check right away?

@thomaseizinger the value of CI for this is IMO close to zero. It's for contributors only and I don't expect it to change anytime soon.


Provided VDB name needs to follow OpenShift resource name convention

when creating a new VDB, a validation needs to be performed to follow the OpenShift resource name convention
see https://docs.google.com/document/d/1N_2YpFdnoRhMInmESEHk01TtCGm9hfursM0MPLQwVSs/edit#bookmark=id.d9goxmxbmbt0

see https://github.com/jboss-fuse/vscode-datavirt/pull/70


Write the JS-code to display the score

The js-code should display the correct score after submitting the answers with the Submit button. The result section---<div class="result" in the html-file should be used to achieve this.
This is the challenge/exercise given in Lesson 68.

This commit has completed the issue.


cmd/go: all.bash times out for go1.17rc1 on OpenBSD/amd64

What version of Go are you using (go version)?

$ go version
go version go1.17rc1 openbsd/amd64

Does this issue reproduce with the latest release?
No problem with the latest stable release go1.16.6
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/rapm/.cache/go-build"
GOENV="/home/rapm/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="openbsd"
GOINSECURE=""
GOMODCACHE="/home/rapm/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="openbsd"
GOPATH="/home/rapm/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/rapm/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/rapm/local/go/pkg/tool/openbsd_amd64"
GOVCS=""
GOVERSION="go1.17rc1"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3926604099=/tmp/go-build -gno-record-gcc-switches"

What did you do?

git pull
git checkout go1.17rc1
cd src
bash ./all.bash

What did you expect to see?
Successful build running all the tests without errors
What did you see instead?

...
ok      cmd/doc 3.452s
ok      cmd/fix 47.487s
go test proxy running at GOPROXY=http://<IP_ADDRESS>:4682/mod
--- FAIL: TestScript (0.07s)
    --- FAIL: TestScript/mod_convert_vendor_yml (0.19s)
        script_test.go:252:
            # We should not create a go.mod file unless the user ran 'go mod init' explicitly.
            # However, we should suggest 'go mod init' if we can find an alternate config file. (0.036s)
            # The command we suggested should succeed. (0.066s)
            > cd ..
            $WORK/test
            > go mod init
            [stderr]
            go: creating new go.mod: module m
            go: copying requirements from vendor.yml
            go: to add module requirements and sums:
                go mod tidy
            > go list -m all
            [context deadline exceeded]
            FAIL: testdata/script/mod_convert_vendor_yml.txt:13: test timed out while running command

    --- FAIL: TestScript/mod_convert_vendor_manifest (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_convert_vendor_json (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_convert_vendor_conf (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_convert_tsv_insecure (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_convert_tsv (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_convert_godeps (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_update_wildcard (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_convert_glockfile (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_convert_glide (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_update_unknown_protocol (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_convert_git (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_update_all (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_convert_dep (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_convert (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_concurrent (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_update (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_clean_cache (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_case_cgo (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_cache_dir (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_tilde (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_case (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_cache_rw (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_test_only (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_build_versioned (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_build_tags (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_build_info_err (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_bad_filenames (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/cover_dot_import (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_race (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_insecure (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_goroot (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_non_pkg (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_legacy (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_issue11307 (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_internal_wildcard (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_insecure_update (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_go_file (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_insecure_redirect (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_insecure_no_longer_supported (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_insecure_env (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_dotfiles (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_insecure_custom_domain (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_dash_t (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_dot_slash_download (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_domain_root (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_custom_domain_wildcard (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_404_meta (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/get_brace (0.00s)
        script_test.go:150: test interrupted during setup: context deadline exceeded
    --- FAIL: TestScript/mod_deprecate_message (0.66s)
        script_test.go:252:
            # When there is a short single-line message, 'go get' should print it all. (0.218s)
            # When there is a multi-line message, 'go get' should print the first line. (0.249s)
            > go get -d multiline
            [stderr]
            SIGQUIT: quit
            PC=0x29617405a m=5 sigcode=0

            goroutine 0 [idle]:
            runtime.thrsleep(0xc000084358, 0x3, 0x0, 0x0, 0xc000084358)
                /home/rapm/local/go/src/runtime/sys_openbsd1.go:15 +0x39 fp=0x23d93c448 sp=0x23d93c420 pc=0x45a539
            runtime.semasleep(0xffffffffffffffff)
                /home/rapm/local/go/src/runtime/os_openbsd.go:112 +0xea fp=0x23d93c4a8 sp=0x23d93c448 pc=0x4354ca
            runtime.notesleep(0xe66758)
                /home/rapm/local/go/src/runtime/lock_sema.go:182 +0x78 fp=0x23d93c4d8 sp=0x23d93c4a8 pc=0x40d9f8
            runtime.templateThread()
                /home/rapm/local/go/src/runtime/proc.go:2385 +0x86 fp=0x23d93c4f8 sp=0x23d93c4d8 pc=0x440826
            runtime.mstart1()
                /home/rapm/local/go/src/runtime/proc.go:1407 +0x97 fp=0x23d93c520 sp=0x23d93c4f8 pc=0x43f197
            runtime.mstart0()
                /home/rapm/local/go/src/runtime/proc.go:1365 +0x66 fp=0x23d93c540 sp=0x23d93c520 pc=0x43f0e6
            runtime.mstart()
                /home/rapm/local/go/src/runtime/asm_amd64.s:248 +0x5 fp=0x23d93c548 sp=0x23d93c540 pc=0x46bee5

            goroutine 1 [chan receive]:
            runtime.gopark(0xa905d0, 0xc000082ef8, 0xe, 0x17, 0x2)
                /home/rapm/local/go/src/runtime/proc.go:366 +0x105 fp=0xc00014db70 sp=0xc00014db50 pc=0x43c665
            runtime.chanrecv(0xc000082ea0, 0x0, 0x1)
                /home/rapm/local/go/src/runtime/chan.go:576 +0x5f7 fp=0xc00014dc00 sp=0xc00014db70 pc=0x407f77
            runtime.chanrecv1(0xc000082ea0, 0x0)
                /home/rapm/local/go/src/runtime/chan.go:439 +0x2b fp=0xc00014dc30 sp=0xc00014dc00 pc=0x40792b
            cmd/go/internal/modget.(*resolver).performPathQueries(0xc0000cc700, {0xb24300, 0xc00009c000})
                /home/rapm/local/go/src/cmd/go/internal/modget/get.go:967 +0x132 fp=0xc00014dc70 sp=0xc00014dc30 pc=0x9476f2
            cmd/go/internal/modget.runGet({0xb24300, 0xc00009c000}, 0xe27a80, {0xc0000b0030, 0x1, 0x1})
                /home/rapm/local/go/src/cmd/go/internal/modget/get.go:294 +0x325 fp=0xc00014dd80 sp=0xc00014dc70 pc=0x941b45
            main.invoke(0xe27a80, {0xc0000b0010, 0x3, 0x3})
                /home/rapm/local/go/src/cmd/go/main.go:216 +0x3b1 fp=0xc00014de50 sp=0xc00014dd80 pc=0x976111
            main.main()
                /home/rapm/local/go/src/cmd/go/main.go:173 +0x9ea fp=0xc00014df88 sp=0xc00014de50 pc=0x97594a
            runtime.main()
                /home/rapm/local/go/src/runtime/proc.go:255 +0x248 fp=0xc00014dfe0 sp=0xc00014df88 pc=0x43c208
            runtime.goexit()
                /home/rapm/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00014dfe8 sp=0xc00014dfe0 pc=0x46e121

            goroutine 2 [force gc (idle)]:
            runtime.gopark(0xa90820, 0xe347a0, 0x11, 0x14, 0x1)
                /home/rapm/local/go/src/runtime/proc.go:366 +0x105 fp=0xc000040fb0 sp=0xc000040f90 pc=0x43c665
            runtime.goparkunlock(...)
                /home/rapm/local/go/src/runtime/proc.go:372
            runtime.forcegchelper()
                /home/rapm/local/go/src/runtime/proc.go:306 +0xc5 fp=0xc000040fe0 sp=0xc000040fb0 pc=0x43c4c5
            runtime.goexit()
                /home/rapm/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000040fe8 sp=0xc000040fe0 pc=0x46e121
            created by runtime.init.7
                /home/rapm/local/go/src/runtime/proc.go:294 +0x35

            goroutine 3 [GC sweep wait]:
            runtime.gopark(0xa90820, 0xe34a60, 0xc, 0x14, 0x1)
                /home/rapm/local/go/src/runtime/proc.go:366 +0x105 fp=0xc0000417b0 sp=0xc000041790 pc=0x43c665
            runtime.goparkunlock(...)
                /home/rapm/local/go/src/runtime/proc.go:372
            runtime.bgsweep()
                /home/rapm/local/go/src/runtime/mgcsweep.go:163 +0xaa fp=0xc0000417e0 sp=0xc0000417b0 pc=0x426b4a
            runtime.goexit()
                /home/rapm/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000417e8 sp=0xc0000417e0 pc=0x46e121
            created by runtime.gcenable
                /home/rapm/local/go/src/runtime/mgc.go:181 +0x75

            goroutine 4 [GC scavenge wait]:
            runtime.gopark(0xa90820, 0xe35280, 0xd, 0x14, 0x1)
                /home/rapm/local/go/src/runtime/proc.go:366 +0x105 fp=0xc000041f80 sp=0xc000041f60 pc=0x43c665
            runtime.goparkunlock(...)
                /home/rapm/local/go/src/runtime/proc.go:372
            runtime.bgscavenge()
                /home/rapm/local/go/src/runtime/mgcscavenge.go:265 +0xf8 fp=0xc000041fe0 sp=0xc000041f80 pc=0x4249f8
            runtime.goexit()
                /home/rapm/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000041fe8 sp=0xc000041fe0 pc=0x46e121
            created by runtime.gcenable
                /home/rapm/local/go/src/runtime/mgc.go:182 +0x8d

            goroutine 18 [finalizer wait]:
            runtime.gopark(0xa90820, 0xe66200, 0x10, 0x14, 0x1)
                /home/rapm/local/go/src/runtime/proc.go:366 +0x105 fp=0xc00003c740 sp=0xc00003c720 pc=0x43c665
            runtime.goparkunlock(...)
                /home/rapm/local/go/src/runtime/proc.go:372
            runtime.runfinq()
                /home/rapm/local/go/src/runtime/mfinal.go:177 +0xc6 fp=0xc00003c7e0 sp=0xc00003c740 pc=0x41b5c6
            runtime.goexit()
                /home/rapm/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00003c7e8 sp=0xc00003c7e0 pc=0x46e121
            created by runtime.createfing
                /home/rapm/local/go/src/runtime/mfinal.go:157 +0x57

            goroutine 21 [runnable]:
            runtime.makeslicecopy(0x9ad600, 0x1f, 0x1e, 0xc0000fd840)
                /home/rapm/local/go/src/runtime/slice.go:36 +0x19c fp=0xc0000b6e38 sp=0xc0000b6e30 pc=0x4520dc
            syscall.ByteSliceFromString(...)
                /home/rapm/local/go/src/syscall/syscall.go:53
            syscall.BytePtrFromString(...)
                /home/rapm/local/go/src/syscall/syscall.go:69
            syscall.Lstat({0xc0000fd840, 0x1e}, 0xc0001739b8)
                /home/rapm/local/go/src/syscall/zsyscall_openbsd_amd64.go:967 +0x85 fp=0xc0000b6ea8 sp=0xc0000b6e38 pc=0x481a25
            os.lstatNolog.func1(...)
                /home/rapm/local/go/src/os/stat_unix.go:46
            os.ignoringEINTR(...)
                /home/rapm/local/go/src/os/file_posix.go:246
            os.lstatNolog({0xc0000fd840, 0x1e})
                /home/rapm/local/go/src/os/stat_unix.go:45 +0x65 fp=0xc0000b6ef8 sp=0xc0000b6ea8 pc=0x4e9a05
            os.Lstat({0xc0000fd840, 0x1e})
                /home/rapm/local/go/src/os/stat.go:22 +0x4d fp=0xc0000b6f38 sp=0xc0000b6ef8 pc=0x4e934d
            os.(*File).readdir(0xc0000aa210, 0xffffffffffffffff, 0x2)
                /home/rapm/local/go/src/os/dir_unix.go:129 +0x7ad fp=0xc0000b7060 sp=0xc0000b6f38 pc=0x4df62d
            os.(*File).Readdir(0xc0000aa210, 0xffffffffffffffff)
                /home/rapm/local/go/src/os/dir.go:41 +0x51 fp=0xc0000b70e0 sp=0xc0000b7060 pc=0x4de811
            io/ioutil.ReadDir({0xc0000fd7a0, 0x1a})
                /home/rapm/local/go/src/io/ioutil/ioutil.go:63 +0x90 fp=0xc0000b7188 sp=0xc0000b70e0 pc=0x589f50
            cmd/go/internal/fsys.readDir({0xc0000fd7a0, 0x1a})
                /home/rapm/local/go/src/cmd/go/internal/fsys/fsys.go:246 +0x39 fp=0xc0000b71e0 sp=0xc0000b7188 pc=0x5d8819
            cmd/go/internal/fsys.ReadDir({0xc0000fd7a0, 0x1a})
                /home/rapm/local/go/src/cmd/go/internal/fsys/fsys.go:270 +0x39c fp=0xc0000b7578 sp=0xc0000b71e0 pc=0x5d8dbc
            cmd/go/internal/fsys.walk({0xc0000fd7a0, 0x1a}, {0xb27188, 0xc000155800}, 0xc0000b77f0)
                /home/rapm/local/go/src/cmd/go/internal/fsys/fsys.go:412 +0x72 fp=0xc0000b7640 sp=0xc0000b7578 pc=0x5da2b2
            cmd/go/internal/fsys.walk({0xc0000c8090, 0x17}, {0xb27188, 0xc000154300}, 0xc0000b77f0)
                /home/rapm/local/go/src/cmd/go/internal/fsys/fsys.go:427 +0x2b8 fp=0xc0000b7708 sp=0xc0000b7640 pc=0x5da4f8
            cmd/go/internal/fsys.Walk({0xc0000c8090, 0x17}, 0xc0000b77f0)
                /home/rapm/local/go/src/cmd/go/internal/fsys/fsys.go:443 +0xdf fp=0xc0000b7770 sp=0xc0000b7708 pc=0x5da6df
            cmd/go/internal/modload.matchPackages.func3({0xc0000c8090, 0x17}, {0x0, 0x0}, 0x2)
                /home/rapm/local/go/src/cmd/go/internal/modload/search.go:58 +0x15d fp=0xc0000b7858 sp=0xc0000b7770 pc=0x8695dd
            cmd/go/internal/modload.matchPackages({0xb24300, 0xc00009c000}, 0xc0000ac8a0, 0xc000152630, 0x1, {0x0, 0x0, 0x0})
                /home/rapm/local/go/src/cmd/go/internal/modload/search.go:127 +0x31d fp=0xc0000b7a50 sp=0xc0000b7858 pc=0x868e7d
            cmd/go/internal/modload.MatchInModule({0xb24300, 0xc00009c000}, {0x7f7ffffe7d6f, 0x9}, {{0x0, 0x0}, {0x0, 0x0}}, 0xc000152630)
                /home/rapm/local/go/src/cmd/go/internal/modload/search.go:187 +0x15b fp=0xc0000b7af8 sp=0xc0000b7a50 pc=0x86a35b
            cmd/go/internal/modget.(*resolver).matchInModule.func1()
                /home/rapm/local/go/src/cmd/go/internal/modget/get.go:653 +0xe8 fp=0xc0000b7bc0 sp=0xc0000b7af8 pc=0x944368
            cmd/go/internal/par.(*Cache).Do(0xc0000cc7d0, {0x9e9480, 0xc0001525d0}, 0xc000053cd0)
                /home/rapm/local/go/src/cmd/go/internal/par/work.go:128 +0x10e fp=0xc0000b7c20 sp=0xc0000b7bc0 pc=0x5f9c6e
            cmd/go/internal/modget.(*resolver).matchInModule(0xc0000cc700, {0xb24300, 0xc00009c000}, {0x7f7ffffe7d6f, 0x9}, {{0x0, 0x0}, {0x0, 0x0}})
                /home/rapm/local/go/src/cmd/go/internal/modget/get.go:652 +0x1c5 fp=0xc0000b7d28 sp=0xc0000b7c20 pc=0x944185
            cmd/go/internal/modget.(*resolver).queryPath.func1()
                /home/rapm/local/go/src/cmd/go/internal/modget/get.go:985 +0x20a fp=0xc0000b7e10 sp=0xc0000b7d28 pc=0x947a8a
            cmd/go/internal/modget.(*query).pathOnce(0xc00012d450, {0x7f7ffffe7d6f, 0x9}, 0xc000053ef8)
                /home/rapm/local/go/src/cmd/go/internal/modget/query.go:264 +0x95 fp=0xc0000b7ed8 sp=0xc0000b7e10 pc=0x950955
            cmd/go/internal/modget.(*resolver).queryPath(0xc0000cc700, {0xb24300, 0xc00009c000}, 0xc00012d450)
                /home/rapm/local/go/src/cmd/go/internal/modget/get.go:975 +0x9d fp=0xc0000b7f30 sp=0xc0000b7ed8 pc=0x94785d
            cmd/go/internal/modget.(*resolver).performPathQueries.func1()
                /home/rapm/local/go/src/cmd/go/internal/modget/get.go:963 +0x74 fp=0xc0000b7f60 sp=0xc0000b7f30 pc=0x947794
            cmd/go/internal/par.(*Queue).Add.func1(0xc0000aa160, 0xc0000939e0)
                /home/rapm/local/go/src/cmd/go/internal/par/queue.go:58 +0x7c fp=0xc0000b7fd0 sp=0xc0000b7f60 pc=0x5f929c
            runtime.goexit()
                /home/rapm/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000b7fd8 sp=0xc0000b7fd0 pc=0x46e121
            created by cmd/go/internal/par.(*Queue).Add
                /home/rapm/local/go/src/cmd/go/internal/par/queue.go:56 +0x25a

            rax    0x58
            rbx    0x23d93c448
            rcx    0x29617405a
            rdx    0x0
            rdi    0xc000084358
            rsi    0x3
            rbp    0x23d93c390
            rsp    0x23d93c388
            r8     0xc000084358
            r9     0xf5ff
            r10    0x0
            r11    0x206
            r12    0x4015e0
            r13    0x7f7ffffe7a88
            r14    0xc0000861a0
            r15    0x46bee0
            rip    0x29617405a
            rflags 0x206
            cs     0x2b
            fs     0x0
            gs     0x0
            [context deadline exceeded]
            FAIL: testdata/script/mod_deprecate_message.txt:8: test timed out while running command

FAIL
FAIL    cmd/go  529.747s
ok      cmd/go/internal/auth    0.202s
...
ok      cmd/trace       4.029s
ok      cmd/vet 74.318s
FAIL
go tool dist: Failed: exit status 1

It looks like the whole test run timed out, and this is the place it happened to time out.
@rialpamu Could you tell us more about this machine? Even the test that succeeded seem to be running much, much slower than I usually se. For example, go test -short cmd/fix passes for me in 1.229s, but here it's 47.487s. Did these tests run significantly faster in 1.16.6?

ALL TESTS PASSED in final go1.17


Group Projects - Milestone 2

High priority (must-fix)

[x] The Dataset has to be either a URL or a relative address to your project directory.
[x] Use heatmap to see the correlation between multiple attributes
[x]  Use a summary section or a conclusion at the end of task 5 to show how well did you answer all the research question

Medium Priority (should-fix)

[x] Please avoid printing the whole dataset
[x] Use one dataset for the whole process
[x] Print top 10 to 20 counties, make the following and other graphs more readable.
"Appearances Per Country over 2014 and 2015",
"Countries sorted by alumni employment" etc.

Low priority (fix if there is time)

[x] Explain how this info will help students to make informed decisions regarding what school to attend

Thank you for the comments and input.  We've successfully fixed most issues that were recognized above.
For issues that were not fixed:

A heatmap was already used when milestone 2 was initially submitted, so perhaps this was missed in the review of milestone.  No other heatmaps were added.
Although our group discussed only printing the top countries in terms of alumni employment, this does not allow the reader to view our intention of including the graph, which is that a large variation exists both between countries, and within countries in terms of alumni employment.  Therefore, we changed the format of the graph to improve readability but did not change the data used.
In order to improve readability as suggested, some correlation graphs were changed to hex bins in order to better show the data.  However, some scatterplots better represented the data than hex bins, so were left as is.  The end result is a mix between hex and scatter plots to best display the data.

Still, using absolute references for the dataset (e.g., r"C:\Users\Amritpal Aujla\data301\course-project-group_6012\data\raw\UniversitiesDataSet.csv" )
Some graphs are not still visible (e.g., countries sorted by alumni employment)
All the graphs are not well explained.

Thank you again for your comments.  I went through to check on the issues that were specified above:

I believe there is still one instance where an individual team member has an absolute reference in the milestone 2 EDA, however, a relative reference is the previous line (I believe that this was intended to be commented out, but was potentially missed).  Regardless, this doesn't exist on any shared items such as the group data analysis.

I'm not exactly sure what you're referring to when you say the graphs are not visible, as I went back to check and they are all present.  Potentially we are using an updated version of matplotlib or seaborn that you don't have, and that's why they aren't showing up when you go to view them?

To my knowledge, the graphs are well explained based on some potential reasons that correlations do (or don't) exist between specific variables and alumni employment.  Additionally, I don't believe they were changed from the initial milestone 2 submission, and no comment was made on them at this point.  More clarification would be greatly appreciated.

I'll plan to come to office hours at 8am tomorrow morning to discuss this with yourself, as it's likely easier/quicker than github issue commenting!! Thanks!


overriden: backoffice: ItemCirculation: Handle empty shelf value

closing in favour of https://github.com/CERNDocumentServer/cds-ils/pull/893


Gradle plugin portal does not show latest version

https://plugins.gradle.org/plugin/io.mateo.cxf-codegen - latest version listed is 1.2.1

As of https://github.com/ciscoo/cxf-codegen-gradle/issues/81, the plugin is no longer published to the Gradle plugin portal. This is also called out in the last release to the portal:

Version 1.2.1 (latest)
Created 23 April 2023.
Gradle plugin to generate code sources from WSDL.
The next major version and subsequent versions will only be published to the Maven Central Repository.

There is nothing I can do (as far as I'm aware) to fix the version mismatch. The portal proxies Maven Central by the way, so even if the portal doesn't list the version, you can still use a recent version if you have not configured Maven Central as a repository for your build environment.


Bump version, fix download_all_cbioportal

Bump pipeline version
Revert change to git clone behaviour in download_all_cbioportal - reverts changes found in 80a2ea7

Should hopefully fix the AWS megatests: https://nf-core-awsmegatests.s3-eu-west-1.amazonaws.com/pgdb/results-4eda7a827f83a9be0470de16331f7105ab48bdd1/pipeline_info/execution_report.html
PR checklist

[x] This comment contains a description of changes (with reason).
[ ] If you've fixed a bug or added code that should be tested, add tests!

[ ] If you've added a new tool - add to the software_versions process and a regex to scrape_software_versions.py
[ ] If you've added a new tool - have you followed the pipeline conventions in the contribution docs
[ ] If necessary, also make a PR on the nf-core/pgdb branch on the nf-core/test-datasets repository.

[ ] Make sure your code lints (nf-core lint .).
[ ] Ensure the test suite passes (nextflow run . -profile test,docker).
[ ] Usage Documentation in docs/usage.md is updated.
[ ] Output Documentation in docs/output.md is updated.
[ ] CHANGELOG.md is updated.
[ ] README.md is updated (including new tool citations and authors/contributors).

Manually triggered AWS megatests on the dev branch after merge: https://github.com/nf-core/pgdb/runs/2447632819?check_suite_focus=true
Let's see if this PR fixes them... 🤞🏻

Results folder: https://nf-co.re/pgdb/dev/results#pgdb/results-e700d967253c5c80985fdcdc48e482dace2bb2d0/ (I guess that the run is still going)


personal-portfolio-website#844 added

Description
Added a personal portfolio website which is made using html and css, no framework is used.
The website is developed responsive.
Fixes:  #844

Type of change

[x] New feature (non-breaking change which adds functionality)

Checklist:

[x] I have made this from my own
[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings

ATTACH SCREEN-SHOTS / DEPLOYMENT LINK

please consider this under GSSoC'22 contribution.

@priyanshi1282 remove empty script.js file, add your project in style_links.json file

script file is removed and project is already added in style_links.json file.

https://deploy-preview-860--cool-front-end-templates.netlify.app/personal-portfolio-website/index.html
Why is the index.html in the images folder!?

index.html is not in image folder 😅.
see this screenshot-


[All] Upgrade to alpha09

@googlebot I consent.

@googlebot I consent.

@nico-gonzalez I've sent you an email, can you check it? Thanks

@googlebot Can you check again?

@googlebot I consent


cannot use env_file with relative path

I cannot reference env_file on stack file when deploying with docker-sdp since it will search on /tmp directory rather than current directory.

Oh. Good catch. We havent used env files yet ourselves so this didnt come up yet.  Want to give it a shot? Otherwise I can take a look next week.

I am afraid that I don't have the expertise to help, but how should I approach it?

can you try with 0.2.8?
pip3 install https://github.com/neuroforgede/docker-stack-deploy/archive/refs/tags/0.2.8.zip

I installed version 0.2.9 and getting this error:
❯ ./deploy.sh
open /tmp/envs/openldap.env: no such file or directory
Traceback (most recent call last):
  File "/home/dim/.local/bin/docker-sdp", line 4, in <module>
    deployer.main()
  File "/home/dim/.local/lib/python3.11/site-packages/docker_stack_deploy/cli/deployer.py", line 274, in main
    docker_stack_deploy()
  File "/home/dim/.local/lib/python3.11/site-packages/docker_stack_deploy/cli/deployer.py", line 226, in docker_stack_deploy
    subprocess.check_call(
  File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/bin/docker', 'stack', 'deploy', '-c', '/tmp/tmph8sm88kf', 'directory']' returned non-zero exit status 1.

Also I see the version number doesn't change on the setup.py file here https://github.com/neuroforgede/docker-stack-deploy/blob/6572534f70de1e8578ef3a5d27e417050a682562/setup.py#L17 I wasn't sure if I got the right version. This is my installation result:
❯ pip3 install --user https://github.com/neuroforgede/docker-stack-deploy/archive/refs/tags/0.2.9.zip
Collecting https://github.com/neuroforgede/docker-stack-deploy/archive/refs/tags/0.2.9.zip
  Using cached https://github.com/neuroforgede/docker-stack-deploy/archive/refs/tags/0.2.9.zip
  Preparing metadata (setup.py) ... done
Requirement already satisfied: pyyaml in /home/dim/.local/lib/python3.11/site-packages (from docker-sdp==0.2.7) (6.0)
Requirement already satisfied: mypy>=0.800 in /home/dim/.local/lib/python3.11/site-packages (from docker-sdp==0.2.7) (1.3.0)
Requirement already satisfied: typing-extensions>=3.10 in /home/dim/.local/lib/python3.11/site-packages (from mypy>=0.800->docker-sdp==0.2.7) (4.6.0)
Requirement already satisfied: mypy-extensions>=1.0.0 in /home/dim/.local/lib/python3.11/site-packages (from mypy>=0.800->docker-sdp==0.2.7) (1.0.0)
Building wheels for collected packages: docker-sdp
  Building wheel for docker-sdp (setup.py) ... done
  Created wheel for docker-sdp: filename=docker_sdp-0.2.7-py3-none-any.whl size=6544 sha256=8570ff44ea7ef369e18428e5c3b272203b173bb1f8c5f8e348e7bae2c6064ddf
  Stored in directory: /tmp/pip-ephem-wheel-cache-wxkyndw9/wheels/73/26/f9/0b20664dd922e881f68f699750228e8c2cf729411e0e145367
Successfully built docker-sdp
Installing collected packages: docker-sdp
Successfully installed docker-sdp-0.2.7

ouch, good catch

see pip3 install https://github.com/neuroforgede/docker-stack-deploy/archive/refs/tags/0.2.10.zip

It works great!!
Thanks for the great work!


[v2] Passing custom context in v2 like in version 1.6.5

Vladimir what about in v2?
following the issue #1950
I tried the same things of version 1.6.5 but in v2 they do not work in the same way...
I need to pass an entire json object from one page to another page without having to request ajax again the same object to read it in every page all the time...
data-context= in the html link as below seems not to work in version 2?
what is the equivalent in v2. for
this
        mainView.router.load({
        reload: true,
        reloadPrevious: false,
        url: 'recipe_details_router.html',
        context: {"title":"Pizza","description":"description mainView.router"},
        });

this
            content = Template7.compile(content)({
            context: response,
            });
            next(content);

or this
< a href="page-context.html" data-context='{"title":"{{context.title}}","description":"{{context.description}}"}' class="item-link">link </ a >
or this
< a href="#" onClick="mainView.router.load({ url: 'page-router.html', context: {title: '{{context.title}}', description: '{{context.description}}' }})" class="item-link">link </ a >
thanks

This works in v2
< a href="page-context.html" data-context='{"title":"{{context.title}}","description":"{{context.description}}"}' class="item-link">link </ a >

This:
< a href="#" onClick="mainView.router.load({ url: 'page-router.html', context: {title: '{{context.title}}', description: '{{context.description}}' }})" class="item-link">link </ a >

->
< a href="#" onClick="mainView.router.navigate('page-router.html', {context: {title: '{{context.title}}', description: '{{context.description}}' }})" class="item-link">link </ a >

This
mainView.router.navigate('recipe_details_router.html', {
        reloadCurrent: true,
        context: {"title":"Pizza","description":"description mainView.router"},
});

Please, use Forum for questions forum.framework7.io

big thanks

I try this before not success but I will try another way to see if working in last version

Please could someone give a full example of how to pass variables between pages using this method. I'm a noob and I'm just not getting it, I've been trying to piece together information from forums but the pre-version 2 example isn't available ... I'm using version 2. Help would be greatly appreciated.


Add security features; Use named port in service

Relates to https://github.com/wiremind/wiremind-helm-charts/issues/148

Currently failing with:

@desaintmartin Do you have an idea which files we need to adapt?

Actually, at the time, I think the "official" Docker image was not available on any hub, and thus the mailu docker image was used. Maybe, if the official one works as expected, we should use it as it seems well-built.

@desaintmartin So I tried to use the official image (clamav/clamav) but seems to be way more complex than the mailu one. We should create a ticket for switching to the official image, for now I secured the exisiting setup.

Thanks again!


Pre- and post-backup commands won't execute

I use resticker to backup my docker volumes on a regular base so I need to stop and restart my docker containers. I wrote bash scripts and put their paths into the appropriate environmental variables.
Restic backups with built-in cron are working like expected but my bash scripts won't start before and after the backup process.
 After manually installing both my scripts worked with docker exec but not within your backup script.
Any suggestions?
BTW: Thank you for writing an otherwise good working restic wrapper!

Same problem.

@hosseinsalemi I have resolved this issue. You have to pull the latest release with mazolino/restic:1.1.1 because latest pulls a much older version.


Rebuild .exe file

Due to the  https://github.com/NotStatilko/tgback/commit/054cafbfc05d2b356703f53cfd140544df2467bb
Yes. BECAUSE OF BRACKETS.
On Windows with executable for now you can use mode 1—2 instead of 1—1

Fixed in tgback3.0 beta(2.1).


cannot connect to win11 ERRINFO_CLOSE_STACK_ON_DRIVER_IFACE_FAILURE

Hello all.
I tried to connect to win11 box using command line xfreerdp /u: /p: /v: and after that failed xfreerdp /u: /p: /cert:ignore /v:.
I get the same error in both cases: ERRINFO_CLOSE_STACK_ON_DRIVER_IFACE_FAILURE
In tcp dump I see TLS is established and disconected soon after it.
I am using FreeRDP version 2.2.0 on Linux Mint 20.3 xfce
From win7 box I'm able to connect to the same win11 box.
I don't really know is this a bug or I'm doing something wrong but any help is welcomed.
Thank you,
Mario

This is a error message sent from the server. Something is ill configured there.
you may have success using /network:auto, /rfx or /gfx options (they trigger the use of a different graphics path server side)


GUI Configuration

There are many configuration options which may only be accessed via execution flag or manual configuration file.
The process of locating/creating the configuration file is difficult for a large portion of users.
Adding all possible configuration options to the GUI would be overwhelming to users and difficult to implement.

Might it make sense to allow direct editing of the clam.conf file, similar to the way in which the clamspeech textbox allows the clamspeech text file to be edited, via the GUI?

Merging the following into this issue:
expose 'split' and 'merge' in GUI- https://github.com/nochowderforyou/clams/issues/50
I am assuming that an an over-arching change to expose config options would also expose these options which are already exposed as runtime flags.


Account creation stalls

Tried to create account kal-el on augur.net, the process stalled. kalel went through with no problem at nearly the same time.

Nevermind, kalel has stalled too, just a bit further in the process. Overloaded nodes?

Both accounts exist and I can connect but they both hold no cash/rep and one of them holds nothing at all

I think so!  We're switching from our hosted nodes to etherscan's API
shortly, which should resolve several problems that pop up when our nodes'
RPC servers are overloaded.
On Mar 20, 2016 11:38 PM, "tyche78"<EMAIL_ADDRESS>wrote:

Nevermind, kalel has stalled too, just a bit further in the process.
Overloaded nodes?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/AugurProject/augur/issues/514#issuecomment-199146101

That explains all.

This should be fixed with the latest account / faucet updates.  Will re-open if The Stall occurs again.


Nuxt3 auto import  and  alias import failed

error info plugin: ts-plugin (2304、2307)
Vue - Official：v2.0.0
System：
版本	Windows 10 专业版
版本号	22H2
安装日期	2023/11/3
操作系统内部版本	19045.4046
体验	Windows Feature Experience Pack 1000.19053.1000.0

The project runs normally but vscode displays abnormally

After some time everything worked fine and I didn't do anything

i found the same error on the other project.

I upgraded to version 2.0.0 of the Vue extension today, and the same issue occurred. When I use alias and auto-imports, VSCode displays errors, though the build process completes successfully.
Somotimes this error disappears, but occurs again.

I'm having the same problem.

Is this still an issue with the latest version and Hybrid Mode - Auto?

@davidmatter
Yes, this issue still persists in the latest version.

Could anyone provide a minimal reproduction?


BBC BASIC for ESP32

Hi Bill,
I’m posting this here as I’m unable to raise an issue against your BBC BASIC project.
I’m interested to know how we could implement the Pico version and the ESP32 version side by side.  There will be a lot of commonalities in them.
I’m not sure how you were planning to get input from the operator, but I was hoping to use the FabGL library to implement the VGA, and PS/2 portions.  I’m also interested in an SD card for external storage and a single line for audio out.
I have been experimenting with building a FabGL computer, as well as purchasing his development board, and I see BBC BASIC as the ideal retro language for a machine based around the ESP32.
I’d also be interested in seeing how this will be implemented on the Raspberry Pi Pico, as I recently purchased one of those to play with.
Cheers,
Mike

I’m guessing that you’re targeting the Pico version towards this board
https://core-electronics.com.au/pimoroni-pico-vga-demo-base.html
This is the hardware that I’m targeting for the ESP32 version
https://www.tindie.com/products/fabgl/fabgl-development-board/


The externalbrowser authenticator must not ask for a password for authentication

Description
The password field is currently required for External browser authentication, however it should be empty and grey.
Only the Username should be required.
DBeaver Version
<IP_ADDRESS>309250303
Operating System
Windows 10
Database and driver
Snowflake db connection
Steps to reproduce

add new connection
select snowflake
select externalbrowser for authenticatior.

Additional context
No response

Hello @TB-HunBoka
Could you please provide further steps?
What do you mean about

The password field is currently required

The password field can be empty for a Snowflake connection.
So, do you have an error during the connection? Could you please provide it?

Hello @LonwoLonwo,
I made a snowflake connection without saving my password and when I tried to connect to the snowflake, I got a popup window to enter my password. I was confused about that, because it was a DBeaver popup not the SSO login page.
The snowflake password was not shared with me, so I used the SSO login everywhere, so I was confused why do the DBeaver asking for my password.
Then, I could login by leaving the password field empty as you mentioned. Also the empty password can be saved, so it will not came up every time I connect to snowflake.
At first the password seemed mandatory for me.
If there isn't any real use case where the password is required for external browser authenticator, then it would make sense for me to disable or gray out the password field.

Maybe this setting is unchecked in your case?

Because DBeaver prompts to add a password during connection only if it is unchecked.
Also, you can try our PRO products; authentication via SSO is implemented there:

You can try it for free: https://dbeaver.com/trial/
For Snowflake, we have the support of materialized views, tasks, pipes viewing, and more. https://github.com/dbeaver/dbeaver/wiki/Snowflake

Hello @LonwoLonwo,
Yes, you are right, the save password checkbox was not checked for me.
However, I was not typing a password in that field, so I tought there isn't anything it can save. I was wrong.
I didn't know about the pro version having a different SSO implemented.
Looking at your screenshot, this is what I was expecting from the externalbrowser, only username and role is requested. No password field.
I misunderstood the externalbrowserm I tought that should be the SSO login, that's why I wasn't expectind any password field in it.
Sorry for any inconvenience.
This ticket can be closed.


Snakefile is run a second time if rule contains run directive

Snakemake version
7.30.1
Describe the bug
When Python code is executed via the run directive, the Snakefile (not the rules within) seems to be executed twice. This can lead to issues, e.g. unexpected results, when some code outside the rules is run twice, or longer run times if that code is computationally expensive.
Minimal example
Snakefile:
print("\nHELLO\n")

rule shell_rule:
    output:
        "shell.txt"
    shell:
        "echo 'test' > {output}"

rule python_rule:
    output:
        "python.txt"
    run:
        with open(output[0], "w") as f:
            f.write("test")

Running the shell_rule:
snakemake -c 1 --force shell_rule

Output:
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job           count    min threads    max threads
----------  -------  -------------  -------------
shell_rule        1              1              1
total             1              1              1

Select jobs to execute...

[Tue Jul 11 17:09:12 2023]
rule shell_rule:
    output: shell.txt
    jobid: 0
    reason: Forced execution
    resources: tmpdir=/tmp

[Tue Jul 11 17:09:12 2023]
Finished job 0.
1 of 1 steps (100%) done
Complete log: .snakemake/log/2023-07-11T170906.938962.snakemake.log

HELLO

Running the python_rule:
snakemake -c 1 --force python_rule

Output:
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job            count    min threads    max threads
-----------  -------  -------------  -------------
python_rule        1              1              1
total              1              1              1

Select jobs to execute...

[Tue Jul 11 17:10:11 2023]
rule python_rule:
    output: python.txt
    jobid: 0
    reason: Forced execution
    resources: tmpdir=/tmp

Building DAG of jobs...
Nothing to be done (all requested files are present and up to date).

HELLO

[Tue Jul 11 17:10:11 2023]
Finished job 0.
1 of 1 steps (100%) done
Complete log: .snakemake/log/2023-07-11T171010.592718.snakemake.log

HELLO

For the shell based rule HELLO is printed only once. This is expected, as there is only one print statement. However, for the Python based rule HELLO is printed twice, indicating that the Snakefile is unexpectedly run twice.

I can confirm this observation (version 7.32.4)

I have run into this myself, I wrote a code to check that the number of GPUs in resources matches the number of GPU IDs provided, however the rule that actually uses GPUs caused the check to be rerun and to fail (because only one GPU was provided to the rule, but the config file still listed two GPU IDs). I suppose there are ways around this, but it can still result in unexpected behaviour.


Ability to show status bar item - shortcut to 'Run Script'.

I was thinking about showing this status bar just after VSCode has been started.
However it requires change in package.json to "activationEvents": ["*"].
With such change status bar item can appear immediately, and "Activate status bar item" trick won't  be needed.
What do you think about such change?

@ppluciennik thanks for this pull request. I've reviewed it and here are my thoughts.
The PR contributes a 'command button' to the status bar that invokes the npm: Run Scripts.
Here are my concerns with using a status bar item like this:

in general, the status bar should be concerned with showing status and not showing command buttons. This would turn the status bar into a tool bar. Imagine, all extensions would add command buttons to the status bar and use it like a toolbar. This would make the status bar messy (it already has UI scalability limits see also this extension.
the status bar item doesn't show additional status of npm script running, e.g., script running in progress etc. Therefore it has minimal value over the command that is provided by the command palette. This cannot justify the early/ '*' activation of the npm extension.

Given these thoughts, I came to the conclusion to not integrate the PR into the extension at this point in time.

I agree with your concerns, however from user's point of view it's more useful to do one click than pressing F1 + typing "npm run script" + Enter.
Extension python does the same thing as my PR (actually I've checked how this extension implemented such functionality).

it's more useful to do one click than pressing F1 + typing "npm run script" + Enter.

let's not forget that there is a keybinding for "npm: Run Script" (and users are asking for having custom keybindings for scripts).

Extension python does the same thing as my PR (actually I've checked how this extension implemented such functionality).

Can you had a screen shot that illustrates this? I can remember that Python shows the testing status in status line and makes the status clickable.

Another

let's not forget that there is a keybinding for "npm: Run Script"
Yes, there is, but not very user friendly :)

Can you had a screen shot that illustrates this? I can remember that Python shows the testing status in status line and makes the status clickable.
Yes it shows status and it's clickable. My solution implements only clickable part (yes I treat status bar as a toolbar).


add more funding to test accounts

fund accounts with 500 ALGO

Codecov Report

Merging #50 (cb886ad) into main (3fd3124) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main      #50   +/-   ##
=======================================
  Coverage   86.14%   86.14%
=======================================
  Files          14       14
  Lines        1191     1191
=======================================
  Hits         1026     1026
  Misses        165      165

Impacted Files
Coverage Δ

src/contracts/methods.py
100.00% <ø> (ø)

tests/conftest.py
100.00% <ø> (ø)

src/utils/testing/resources.py
72.54% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 20092dd...cb886ad. Read the comment docs.


Look into azure-pipelines self-generated matrix from a script

https://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#set-a-multi-job-output-variable
The PRs I have open now are a bit of a mess because I haven't had time to sort out the merge conflicts in the generated artifacts.
It's not good to commit generated artifacts.
But if I implement the above linked approach, we should be able to get rid of the generated yaml workflow that must be done pre-commit.

I don't know how I didn't see this part before!
https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#multi-configuration
I wonder if they recently added this documentation.