Error loading FinOps Hubs source in Cost Summary and Committment Discounts reports

🐛 Problem
Follow the deployment guide on the FinOps Toolkit Documentation page to deploy my FinOps Hubs resources, and I've set up the FOCUS cost export as suggested. When downloading the Power BI report templates and setting up the Power Query parameter, i.e storageUrlForPowerBI, I get an error when trying to refresh the PowerBI datasets.
👣 Repro steps

Follow the deployment guide on the FinOps Toolkit Documentation page.
Deploy FinOps Hubs using template
Setup FOCUS Cost exports for subscription. Called my export finops-hubs-export, and connected it to the storage account created by the FinOps Hubs deployment. As a created the FOCUS export for one of my subscriptions, I set the target directory to subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Run export manually to get some initial data
When the export has completed, the msexports_transform pipeline in the ADF instance is triggered and completes successfully.
After successful pipeline run, ingestion container in the storage account have been populated with a parquet file containing cost data.
Navigating to the FinOps Hubs resource group, then to deployments, and then to the deployment called hub. From the outputs pane I retrieve the storageUrlForPowerBI value.
Download the Cost Summary Power BI report template.
Open the Power Query editor to update data source parameter.
Press close and apply, which triggers a refresh when redirected back to Power BI desktop.
I am prompted with a login-popup, where I enter my organizational account details to authenticate towards the storage account (My organizational account has the Storage Blob Data Contributor Azure RBAC role). After successful login, I press connect.
After attempting to refresh the data, the CostDetails dataset fails to refresh with error We cannot convert the value null to type Table.

🤔 Expected
After completing repro steps 1-12 I would expect the datasets to refreshed without any errors.
Made some other observations that may imply that there are some bugs in the ADF pipeline or the Power Query functions in the Power BI template.

The ADF pipeline activity setting the metric variable sets it to amortizedcost, where it should be focuscost (but this was easily configurable). See screenshot 1.
The ADF pipeline activity setting the Date variable sets it to name-of-focus-export, which in turn sets the destination folder path to ingestion/subscriptions/name-of-focus-export/amortizedcost. See screenshot 2.
Inspecting the ftk_Hub Power Query function, the folder path is parsed to extract both dataset and month, but given the folder structure produced by the ADF extract pipeline, the folder path parsing would yield erroneous results, causing the dataset refresh failure.
Inspecting the storage account containers, the cost export produces files in the following directory: msexports/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/name-of-focus-export/20240101-20240131/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/.
The ADF transform pipeline produce transformed cost data in the following directory: ingestion/subscriptions/name-of-focus-export/amortizedcost/.
Looking at the code snippet in screenshot 3, the Dataset variable would seem to yield name-of-focus-export, whereas the Month variable would seem to yield amortizedcost.

📷 Screenshots
Screenshot 1

Screenshot 2

Screenshot 3

ℹ Additional context
🙋♀ Ask for the community

We could use your help:

Please vote this issue up (👍) to prioritize it.
Leave comments to help us solidify the vision.

@flanakin I made the following changes to make the Power BI refresh work:

Updated metric variable in ADF pipline activity to be focuscost.
Updated dynamic expression in ADF pipeline activity that sets the Date variable from @split(pipeline().parameters.folderName, '/')[sub(length(split(pipeline().parameters.folderName, '/')), 4)] to @substring(split(pipeline().parameters.folderName, '/')[sub(length(split(pipeline().parameters.folderName, '/')), 2)], 0, 6)
Updated the ftk_Hub Power Query function in the CostSummary report template. Updated lines 40-45 to

    FilterRowsByDate =         if start = null and end = null then tbl         else Table.SelectRows(tbl, each             (start = null or DateTime.From([ChargePeriodEnd]) >= DateTime.From(start))             and (end = null or DateTime.From([ChargePeriodStart]) < DateTime.From(end))         ),

Same error for me.
Also maybe of note the data source selection is missing the CM connector option. If inserted manually, this results in the same error anyway.

@tobiasny Can you confirm what version of hubs you have deployed? Look in storage at config/settings.json. Also, note that after upgrading to hubs 0.2, you'll need to delete the historical data and re-export it using FOCUS. We'll document this upgrade process more clearly...

@psilantropy The connector is not supported in the Cost summary and Commitment discounts reports anymore. We ran into perf issues when trying to do both in the same report, so we split them out to simplify. Use the new CostManagementConnector.pbix report.

@flanakin I can confirm that from the settings.json that I've deployed FinOps Hubs v0.1.2, thanks for clarifying. I used the Deploy to Azure button in the documentation, so I guess that link points to the older FinOps Hubs version and not the new one.

@psilantropy The connector is not supported in the Cost summary and Commitment discounts reports anymore. We ran into perf issues when trying to do both in the same report, so we split them out to simplify. Use the new CostManagementConnector.pbix report.

I was just trying anything.
Same problem as tobiasny anyway. 0.1.2 when I intended to deploy 0.2 but i used the docs and deploy. This might be the source of all issues.
@flanakin You have your Toolkit v0.2 Latest link, then;
New in 0.2, FinOpsHubs. This links you right back to here https://microsoft.github.io/finops-toolkit/hubs where we probably deploy the old version via the deploy button.
Is there an upgrade process or wipe/redo?


Retrieve collations from the schema (and refactor the column info structures)

Requires #813 because of a minor merge conflict; otherwise independent.

Supersedes by #814 due to not running CI here for some reason.


Pop up window not covering elements with high z-index

Describe the bug
When calling a pop up window some of the underling icons still show in front of the pop-up.
I have a picture entity that has elements with z-index defined, and sometimes they show up in front of the pop-up:
This happens with elements with a z-index of 5, i found out it varies with the zoom or with the operating system/browser used.
Changing the z-index to 1 seems to fix it, at least with any reasonable zooming in.
To Reproduce
Steps to reproduce the behavior:

Create a picture elements card with some icons with a z-index of 5
Call a bubbles pop-up
[zoom in on certain browsers]

Expected behavior
The pop up should stay in front on anything else
Screenshots

This is with normal (100% zoom on Chrome, 4K resolution screen): there are icons blurred as it should be (in the bottom left corner of the image)

This happens when zooming in in the same setup as above: some icons pierce thru the pop-up and show up in front of everything.
The same thing happens without the need to zoom on the iOS app (standard HA app, iPAD pro). Everything is fine on Android official app on S23 Ultra
Additional context
The icons are still fully functional: I can click on it and the tap_action get executed, this can be annoying if they are in front of a button in the pop-up.
Thank you! 🍻

Hi and thank you for the feedback, does it happens when your pop-ups are placed after all your other cards?

Yes, as per instructions the vertical-stacks with the popups are after all the other cards.
No other particular option is selected, this is the YAML of the pop-up card:
type: custom:bubble-card card_type: pop-up hash: '#antifurto-casa' show_header: true scrolling_effect: false show_name: false show_icon: true icon: mdi:alarm-light name: Antifurto casa
Just notice that setting z-index to 1 fixed this for me, in fact I have no need to keep the z-index that high I just gave myself some space for the future. I think it's a quite specific user case.

In that case this is expected, the pop-ups have a z-index of 5 as well, higher z-index cover some default HA elements like the more info panel and I can't change that for that reason.
I will close this because there is nothing that I can do without breaking essential features.


Update versions: k8s and CRI-O

This PR contains 2 commits to update the releases of dependencies we use for testing:

CRI-O to 1.0.4
Kubernetes to 1.8.3

LGTM

lgtm

lgtm


Excel can't save , and read wrong data

Hi everyone.
I am newbie in PowerShell, I only try to edit cell A1 in Excel file and check cell B1 have Formula connect with cell A1 , I want get value B1 when I change value in A1
Problem is , I can change value in Cell A1 but the value in B1 not change even I save or save as or close excel for autosave,
but it still not get correct value in B1

Hi,
I faintly remember having had a similar issue, which was resolved when I used the switch -calculate when closing the excel package, see also https://github.com/dfinke/ImportExcel/blob/master/mdHelp/en/close-excelpackage.md#-calculate

Thank you so much , it work,  I have other question, Could I calculate without closing the excel then i can get value :D, it will be great, because I need use excel calculate alot and get data from excel.
I try to build automation excel with PowerShell
Thank you so much

I have other question , Sorry I ask alot question , do we have document about those functions or syntax like picture.
If have those document I can learn faster
Thank you so much.

Thank you so much , it work, I have other question, Could I calculate without closing the excel then i can get value :D, it will be great, because I need use excel calculate alot and get data from excel. I try to build automation excel with PowerShell Thank you so much

Alright,  the diffcult questions! :-)
$xl=Open-ExcelPackage -Path .\test.xlsx
$ws=$xl.Workbook.Worksheets['ABC']
$ws.cells["D1"].Value=23.998
#Formula in  E1: $ws.cells["E1"].formula="+d1*2"
[OfficeOpenXml.CalculationExtension]::Calculate($ws.cells["E1"])
$ws.cells["E1"].value
To answer your other question:

Search the web
https://github.com/dfinke/ImportExcel/tree/master/mdHelp/en
And if you need to dig deeper: https://epplussoftware.com/docs/5.3/api/OfficeOpenXml.html
Try & Error (without going ballistic) :-)

Thank you so much, it work :D

Thank you @TWalijew was heading off to test the Calculate method. This looks like a good candidate for the Examples folder 😉


PRODUCTION 2020?

Have a PRODUCTION 2020?

You can download the latest PRODUCTION from Habbo.

Would you have any programs or scripts to generate a "catalog_items" in the database format?

the closest i have is this: https://github.com/higoka/sql-creator-layout
its not really possible to automate this since the items have different width, length, price etc. so you would need to insert them manually.

Could you provide me with an updated 2020 swf and emulator link?

check out arcturus morningstar: https://git.krews.org/morningstar/Arcturus-Community
this are the official swfs: https://git.krews.org/morningstar/morningstar-catalogue

Sorry to bother. Could you provide me with a compatible database?

the database is included in the emu

Im going to close this issue since its not related to habbo-downloader.


python27 software Permission denied issues

Hello I am hearing impaired and uses subtitles or closed captioned (CC) for videos  watching. Some videos don't have subtitles or CC on them. I found this interesting pyhton 27 software that automatically makes subtitles in videos.  I understand that the video format for this to work is in MP4 only.  I thought I had everything I need to have to make this video subtitles to work.  However, according to the Windows command process, I have found the following errors:
Using windows 10 operating system 32-bit and Firefox browser.
Python27 scripts autosub flac permission denied
C:\Users\d\Desktop>C:\Python27\python.exe C:\Python27\scripts\autosub_app.py -C 2 -S en -D en C:\Users\d\Desktop\EnglishObama.mp4
c:\users\d\appdata\local\temp\tmpyubnxz.flac: Permission denied| ETA:  --:--:--
Exception in thread Thread-3:
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 801, in __bootstrap_inner
self.run()
File "C:\Python27\lib\threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "C:\Python27\lib\multiprocessing\pool.py", line 389, in _handle_results
task = get()
TypeError: ('init() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())
c:\users\d\appdata\local\temp\tmp4boeti.flac: Permission denied
c:\users\d\appdata\local\temp\tmpo75t4d.flac: Permission denied
c:\users\d\appdata\local\temp\tmpwnksga.flac: Permission denied
c:\users\d\appdata\local\temp\tmpqg3ifz.flac: Permission denied
c:\users\d\appdata\local\temp\tmpdswmie.flac: Permission denied
c:\users\d\appdata\local\temp\tmpbf5ixl.flac: Permission denied
c:\users\d\appdata\local\temp\tmpau41p2.flac: Permission denied
c:\users\d\appdata\local\temp\tmpzxgemy.flac: Permission denied
c:\users\d\appdata\local\temp\tmpmsuagn.flac: Permission denied
c:\users\d\appdata\local\temp\tmpe_64dy.flac: Permission denied
c:\users\d\appdata\local\temp\tmptsapsp.flac: Permission denied
c:\users\d\appdata\local\temp\tmpdwdl8y.flac: Permission denied
c:\users\d\appdata\local\temp\tmpzp1okl.flac: Permission denied
c:\users\d\appdata\local\temp\tmp1mp76j.flac: Permission denied
c:\users\d\appdata\local\temp\tmpnhiafl.flac: Permission denied
c:\users\d\appdata\local\temp\tmppkqsqc.flac: Permission denied
c:\users\d\appdata\local\temp\tmppcfouo.flac: Permission denied
c:\users\d\appdata\local\temp\tmpwtpw2m.flac: Permission denied
c:\users\d\appdata\local\temp\tmp6puagf.flac: Permission denied
c:\users\d\appdata\local\temp\tmpubgdua.flac: Permission denied
c:\users\d\appdata\local\temp\tmpxuftxp.flac: Permission denied
c:\users\d\appdata\local\temp\tmp9_geaj.flac: Permission denied
c:\users\d\appdata\local\temp\tmp9qu6ej.flac: Permission denied
c:\users\d\appdata\local\temp\tmp5lemmm.flac: Permission denied
c:\users\d\appdata\local\temp\tmprzifh5.flac: Permission denied
c:\users\d\appdata\local\temp\tmp_ndctx.flac: Permission denied
c:\users\d\appdata\local\temp\tmp1srgv_.flac: Permission denied
c:\users\d\appdata\local\temp\tmpyuolrh.flac: Permission denied
c:\users\d\appdata\local\temp\tmpk60oep.flac: Permission denied
c:\users\d\appdata\local\temp\tmpr8pma8.flac: Permission denied
c:\users\d\appdata\local\temp\tmpv_4z00.flac: Permission denied
c:\users\d\appdata\local\temp\tmp0unmgn.flac: Permission denied
I am a newbie at this. I followed the comments in regards to these issues. However, I am lost as to how to repair/resolve the problems above.. I would like to know how to resolve these issues to make this subtitles work using simple understanding step by step process.  Any help out there is greatly appreciated.

I finally resolved the issue. I did not saved the modiied script (txt. file) in the autosub correctly to the py file format . I was abled to play the sample file with autosubs and works great. Thanks for all the help in this forum.


on startup the serverType value should be written to the log

I was just debugging an app configured for SAS9 that was calling Viya
Turned out the serverType was defined as servertype
We should write message to the log on adapter instantiation to display key values (such as servertype, serverUrl etc) for faster debugging, eg:
The SASjs Adapter has been instantiated!  The following default values are set:
  serverUrl:
  serverType:
  debug:
  (if Viya we can show the extra variables)
To override the defaults, simply pass an updated config in the request method.

@allanbowe Should we only do this if debug is true?

I'd say no - this will typically happen only once (on app startup) and the fact that debug is not set can be the cause of some issues

Closing this - now we use the adapter extensively in the CLI, where it is instantiated with each request, to add this would result in too much logging


feat: support adding items with custom attributes.

[x] PR title and description conform to Pull Request guidelines.

Description of changes:

support adding items with custom attributes.

NOTE: This is a breaking change, only pipelines from version 1.1+ can handle items with custom attribute.
How did you test these changes?
added test case to cover all item error code.
Documentation update required?

[ ] No
[x] Yes (Please include a PR link for the documentation update)

General Checklist

[x] Added Unit Tests

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report
Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (f74a4c9) 87.00% compared to head (0c2a86c) 87.06%.

Additional details and impacted files
@@             Coverage Diff              @@
##               main      #45      +/-   ##
============================================
+ Coverage     87.00%   87.06%   +0.05%
- Complexity      420      443      +23
============================================
  Files            40       42       +2
  Lines          1485     1592     +107
  Branches        129      141      +12
============================================
+ Hits           1292     1386      +94
- Misses          121      131      +10
- Partials         72       75       +3

Files
Coverage Δ

...aws/solution/clickstream/AWSClickstreamPlugin.java
84.53% <100.00%> (+0.32%)
:arrow_up:

...are/aws/solution/clickstream/ClickstreamEvent.java
100.00% <100.00%> (ø)

...s/solution/clickstream/client/AnalyticsClient.java
94.56% <100.00%> (ø)

...oftware/aws/solution/clickstream/client/Event.java
100.00% <ø> (ø)

...aws/solution/clickstream/ClickstreamAnalytics.java
82.92% <0.00%> (-2.08%)
:arrow_down:

.../aws/solution/clickstream/client/EventChecker.java
98.13% <98.13%> (ø)

...ware/aws/solution/clickstream/ClickstreamItem.java
83.33% <83.33%> (ø)

...ws/solution/clickstream/client/AnalyticsEvent.java
77.91% <71.42%> (-1.26%)
:arrow_down:

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Feature/PYPE-762 multi root

Integration of multi root

At the time of PR creation it is more draft and NOT final version!!!
There are many parts I'm sure I forgot about, but need to test it at this stage to know where exatly these parts are.
Please let in comments what do you think. (I'll let you what I think).

Please make comments inside the code to easier see what are you talking about.

Changes

~/pype-config/system/ was removed

storage.json has changed to roots.json in anatomy
core.json was for setting PYPE_STUDIO_CORE_PATH/MOUNT which as was found is not necessary

Storage class was removed from pypeapp
for backwards compatibility there are 2 possiblities.
1.) set default roots.json without anatomy keys same as storage.json was.
2.) copy exact the same json as mentioned in (1) into each project overrides.

EXAMPLE OF BACKWARDS COMPATIBLE roots.json
{
    "windows": "P:/projects/publish",
    "linux": "/mnt/share/projects/publish",
    "darwin": "/darwin/path/publish"
}

Ftrack action Prepare Project  asks for filling of each root key set in default.yaml

there is logic how it is pre-filled which I can explain if anybody asks

Anatomy class is now a wrapper for Templates and Roots.

If you are using anatomy for creating paths, you should not care about roots.

Not really true inside avalon, but most use cases of avalon are inside host where project is already specified so no worries now.

this PR requires to NEVER EVER use environments for getting ROOT in code, always MUST use Anatomy, Roots object for that.
Roots object has find_root_template_from_path which can return path with {root[root type]} if project roots are matching entered path.

similar feature should be added to filled template by anatomy. Filled template should have rootless attribute where path without root key will be stored. With this it may be easier to achieve partially filled paths.

Testing

check TODO/Question below
create new project (if you don't want to break your other project)
copy roots.json to project override or use prepare action as explained above
run application
publish
load
publish to farm
watch errors

TODOs/Questions/Issues

publish paths are not correct on farm (I am 99% sure)

this is more question than TODO, how to do it
I had idea to have partially filled paths (without root) until integration happens

That is hard to do, each host has many, many parts where path are created

Ftrack action Create project folders do not use Anatomy for creating folders right now and folders are created for each root defined in project's template, which may cause issues in future.
thumbnails_root was kept as it is because we exect to store in one place. But it easy to replace {thumbnail_root} with key anybody wants

Obviously this introduces quite a problem if someone didn't use project override on anatomy before and now they change to multiroot. It pretty much invalidates all the old publishes, but oh well, there's not much to be done about that other than careful deployment.

Maybe we can create upgrade script that will crawl over publishes and fixes their paths?

Maybe we can create upgrade script that will crawl over publishes and fixes their paths?

that wouldn't work. the publishes are ok, or we ok at the time of publish, however if you specify 2 roots, they fill in dynamically when loading hence no finding the originally published file. Now if you set your root_publish to the same path as the the originally used single root, that all with keep working. It's not a technical issue, but more of a philosophical.
But I've tried and there is actually a minor bug. If I change project from single to multiroot. Even if I set both roots to the same path the it was originally. get_representation_path doesn't correctly fill it. There is } that stays hanging. I'll make reproducible a post details tomorrow.

We have some conflics. @iLLiCiTiT could you please have a look? I wanna make a release candidate on Friday for 2.9.
@antirotor Could you please give this a shot on-site with a mixed farm by then?

We have some conflics. @iLLiCiTiT could you please have a look? I wanna make a release candidate on Friday for 2.9.

resolved

# Traceback (most recent call last):
#   File "K:\dev\pype-setup\repos\avalon-core\avalon\tools\workfiles\app.py", line 851, in _on_task_changed
#     files.set_asset_task(asset, task)
#   File "K:\dev\pype-setup\repos\avalon-core\avalon\tools\workfiles\app.py", line 457, in set_asset_task
#     session = self._get_session()
#   File "K:\dev\pype-setup\repos\avalon-core\avalon\tools\workfiles\app.py", line 473, in _get_session
#     task=self._task)
#   File "K:\dev\pype-setup\repos\avalon-core\avalon\pipeline.py", line 1112, in compute_session_changes
#     changes["AVALON_WORKDIR"] = _format_work_template(template, _session)
#   File "K:\dev\pype-setup\repos\avalon-core\avalon\pipeline.py", line 1170, in _format_work_template
#     "root": registered_root(),
#   File "K:\dev\pype-setup\repos\avalon-core\avalon\pipeline.py", line 768, in registered_root
#     Session.get("AVALON_PROJECTS") or ""
#   File "C:\Program Files\Autodesk\Maya2019\bin\python27.zip\ntpath.py", line 419, in normpath
# AttributeError: 'dict' object has no attribute 'startswith'
Using existing QApplication..
# Error: AttributeError: file C:\Program Files\Autodesk\Maya2019\bin\python27.zip\ntpath.py line 419: 'dict' object has no attribute 'startswith' #

when running Loader from Maya

This hits bug in Acre if multiroot path string is in environment variable, like:
{root[work]}/foo/bar/baz

It is fixed in BigRoy/acre#8


Remove ambiguous

Remove ambiguous = infer parameter

Codecov Report
Patch coverage: 100.00% and no project coverage change.

Comparison is base (96a78d5) 97.91% compared to head (24a16af) 97.91%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #105   +/-   ##
=======================================
  Coverage   97.91%   97.91%
=======================================
  Files           9        9
  Lines         576      576
=======================================
  Hits          564      564
  Misses         12       12

Impacted Files
Coverage Δ

cluster_experiments/random_splitter.py
99.15% <100.00%> (ø)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


bump version dependency for dm frameworks

Pull in latest frameworks for draft g9 hosting+software questions.

:+1: once tests pass


Use thumbnail_url for thumbnail generation when present

Problem

Some providers (like SMK) may link to quite large images in their image_url. This can cause unacceptably slow response times or even timeouts when generating thumbnails via our thumbnail service.
Description

We already have a thumbnail_url available on the Image model which we can use.
When the thumbnail_url is available, we should send this to the thumbnail service instead of the image_url. This should be a small change here:
image_url = image.thumbnail or image.url

(using the serialized field names)
Additional context

https://github.com/WordPress/openverse-catalog/issues/698 tracks updating the SMK provider script in the Catalog to populate thumbnail_url with a link to a smaller image size.
Implementation

[ ] 🙋 I would be interested in implementing this feature.

Following @obulat's request of evaluating the stored thumbnails, I ran a couple of queries on the API database to investigate.

See the providers with thumbnails
SELECT DISTINCT(provider) FROM image WHERE thumbnail IS NOT NULL ORDER BY provider ASC;

+--------------------+
| provider           |
|--------------------|
| 500px              |
| eol                |
| flickr             |
| iha                |
| met                |
| nappy              |
| phylopic           |
| rawpixel           |
| rijksmuseum        |
| sciencemuseum      |
| sketchfab          |
| smk                |
| stocksnap          |
| thingiverse        |
| thorvaldsensmuseum |
| waltersartmuseum   |
| wordpress          |
+--------------------+
SELECT 17
Time: 1950.363s (32 minutes 30 seconds), executed in: 1950.335s (32 minutes 30 seconds)

Following this result, I was curious to know how many thumbnails we have for each provider, omitting '500px', 'eol', 'iha', 'thorvaldsensmuseum' and 'waltersartmuseum' providers because those are disabled at the moment.
Thumbnails availability by (active) provider
SELECT provider,
    COALESCE(SUM(CASE WHEN thumbnail IS NOT NULL THEN 1 ELSE 0 END), 0) AS thumbs_available,
    COALESCE(SUM(CASE WHEN thumbnail IS NULL THEN 1 ELSE 0 END), 0) AS thumbs_not_available
FROM image
WHERE provider IN (
    'flickr', 'met', 'nappy', 'phylopic', 'rawpixel', 'rijksmuseum', 'sciencemuseum', 'sketchfab', 'smk', 'stocksnap', 'thingiverse', 'wordpress'
)
GROUP BY provider
ORDER BY provider ASC;

+---------------+------------------+----------------------+
| provider      | thumbs_available | thumbs_not_available |
|---------------+------------------+----------------------|
| flickr        | 497009314        | 911578               |
| met           | 128707           | 234675               |
| nappy         | 2059             | 0                    |
| phylopic      | 3145             | 1011                 |
| rawpixel      | 26239            | 136508               |
| rijksmuseum   | 30000            | 0                    |
| sciencemuseum | 82558            | 12143                |
| sketchfab     | 37872            | 0                    |
| smk           | 106651           | 1416                 |
| stocksnap     | 35436            | 1625                 |
| thingiverse   | 32659            | 0                    |
| wordpress     | 2518             | 3202                 |
+---------------+------------------+----------------------+
SELECT 12
Time: 2324.308s (38 minutes 44 seconds), executed in: 2324.287s (38 minutes 44 seconds)

The 'nappy', 'rawpixel', 'stocksnap' and 'wordpress' provider were recently added so I assume their thumbnails are acceptable. For the rest, I queried some samples and gathered some notes.
Old providers with thumbnails stored
The size shown is only for the sample at the right, it does not apply to all of the supplier's.

provider
Size
Sample thumbnail

flickr
240X180 (small)
https://live.staticflickr.com/4397/36340706826_ee1c70429d_m.jpg

met
600x437
https://images.metmuseum.org/CRDImages/dp/web-large/DP818514.jpg

phylopic
N/A
Not working

rijksmuseum
200x165 (small)
https://lh3.ggpht.com/Nsf8GjZnb8bDQJF2L7hzVc1YsBmG1Sf3rRPVC_ftd2GkXJRIckMvy8zoRBaXRXXnNOe8NUe0bLEMjxDnMpUthfmHlA=s200

sciencemuseum
320X256
https://coimages.sciencemuseumgroup.org.uk/images/516/194/large_thumbnail_1990_5131_11184__0001_.jpg

sketchfab
256x144 (samll)
https://media.sketchfab.com/models/01758a58e868439d89b4c7e4c97afc98/thumbnails/5bca6d57c97f4b32a95ecfb35a3a1d9a/b54599a7e5614362b0e87fa7355c87f1.jpeg

smk
400x390
https://iip.smk.dk/iiif/jp2/kksgb22133.tif.jp2/full/!400,/0/default.jpg

thingiverse
333x250
https://cdn.thingiverse.com/renders/a2/08/4a/a6/a2/cc096c76061794fdf9c525971ab59948_display_medium.jpg

The only thumbnails not working are from 'phylopic', so I'll omit them. For the rest, some are small but since we don't have specific quality requirements at the moment I'll assume they are fine as long as they are online.

Thank you for this amazing investigation, @krysal!

The only thumbnails not working are from 'phylopic', so I'll omit them. For the rest, some are small but since we don't have specific quality requirements at the moment I'll assume they are fine as long as they are online.

I would prefer not to use the small thumbnails (flickr, rijksmuseum, sciencemuseum in the table) because when we discussed this issue (https://wordpress.slack.com/archives/C02012JB00N/p1678203055330969), the lowest width mentioned seems to have been 400px.
Having said that, Flickr might be OK to use because I suspect that the smaller thumbnails are from older images, and newer ones will have larger thumbnails available soon.

@obulat how would you suggest proceeding in that case? Set the thumbnail to null for all other providers beyond Flickr that don't have at least 400px to a side?

I'd like to leave the code for using thumbnails without exception and address the small thumbnails in consequent issues. As I said, since they're working, it should be okay, and if it's important to have better thumbnail quality we can rise #905's priority and the sub-issues that may come from there.

From a frontend perspective I really wouldn't want to go live with thumbnails below the 400px size we identified. The thumbnails are a user's first glimpse at potential results and if they're pixelated and blurry it's going to discourage them from ever using Openverse again. Using thumbnails which are smaller then we currently use feels like a big usability reduction that I don't think we should ship.

@obulat how would you suggest proceeding in that case? Set the thumbnail to null for all other providers beyond Flickr that don't have at least 400px to a side?

On the catalog side, I can think of two approaches for providers with no acceptable thumbnail available: either use the same URL as the main image or set the thumbnail to None. The first approach would mean that the code in the API is the same for all providers: take the image.thumbnail (if available, during the transition stage). The second approach would mean that we would use image.thumbnail if it's available (but we set the thumbnail to null for the existing images where the thumbnails are too small), and image.url otherwise.
Based on what we do in the catalog, the API would either always use the image.thumbnail (the first approach above) or use image.thumbnail for some providers and image.url for others. I guess we could have a configuration set for providers with an acceptable thumbnail, and check against it before falling back to image.url.
I have one more question about this issue. Do we want to thumbnail a thumbnail?. If we already have a thumbnail URL in the database, do we want to generate our own thumbnail out of it? Would this adversely affect the quality of the resulting image? What are the reasons for not using the provider's thumbnail link directly? Provide the Openverse cache layer for it to prevent dead images for the cached items?

I'd like to leave the code for using thumbnails without exception and address the small thumbnails in consequent issues. As I said, since they're working, it should be okay, and if it's important to have better thumbnail quality we can rise #905's priority and the sub-issues that may come from there.

Considering the implications of bad thumbnail quality on users' perceptions, I would prefer for us to go in another direction: start with only the providers that have good thumbnails (SMK and Met), and add other thumbnails when the other providers' thumbnail quality improves.

My only goal is to prevent us from shipping a noticeable visual regression to users. In the case of Openverse, 90% of our users are on desktop:

On a typical laptop screen the dimensions of each image thumbnail are around ~2-300px. When we factor in contemporary high DPI displays that means 4-600px is an ideal size. Anything smaller is going to stretch. Here's an example with a 200x200 image of a cat which at its default size is sharp and high quality:

I don't feel qualified to manage the technical approach to how to fix the PR. Perhaps deleting all non-viable thumbnail urls will be the best.

Would it be possible for us to test this out on staging? It's hard to assess how this might impact the search view since we're presently generating our own thumbnails for all of these providers.
– @AetherUnbound

Good idea! As @sarayourfriend will perform tests in staging with near-production data size, the timing couldn't be more perfect. We just need to merge #898 before.

I think given whatever we discuss here, we'll also need to update our ingesters (or make note for future ingesters) that we should only record a thumbnail if it meets a minimum size criteria.
– @AetherUnbound

Agree, I can do that after @zackkrida or @obulat (or anyone else willing to) evaluate the use of the stored thumbnails in staging. and delete the bad ones by providers if needed. Hope this gives us more confident in this solution.


Daily Actions - Errors

Describe the bug
So i ran the daily activites task after I had done both my free summon and pet adoption.
Issue 1: When it tried to do it, it couldn't find the covenent summon button, becuase there are currently 4 banners and it did not scroll down. It somehow ended up summoning on the tenbria banner instead.
Issue 2: For the pet adoption, I had no tickets. So it just kept going with the script without exiting to lobby (trying to click on guild etc). Could we introduce a check to see that a free summon is available and then proceed?
Steps to reproduce
Perform daily summon and pet adoption, then run the bot.
Additional context
No response
Validations

[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] The provided reproduction is a minimal reproducible example of the bug.

Reopen of #54


HIVE-24310: Allow specified number of deserialize errors to be ignored

What changes were proposed in this pull request?
Allow specified number of deserialize errors to be ignored

Why are the changes needed?
Sometimes we see some corrupted records in user's raw data,  like one corrupted in a file which contains over thousands of records, user has to either give up all records or replay the whole data in order to run successfully on hive, we should provide a way to ignore such corrupted records.

Does this PR introduce any user-facing change?
No

How was this patch tested?
unit tests

@sershe-apache @prasanthj  any thoughts or comments on this? thank you!

@belugabehr @kgyrtkirk any thoughts or comments on this pr? thanks in advance!

Hi @abstractdog  @mmccline, any thoughts or comments here?
Thanks.


bs_seeker2-align.py and Cannot open input file

Hi,
When I run bs_seeker2-align.py, the following error was reported:
python /home/shg047/software/BSseeker2/bs_seeker2-align.py --help
File "/home/shg047/software/BSseeker2/bs_seeker2-align.py", line 136
print "[Error] Cannot open input file : %s" % read_file
^
SyntaxError: Missing parentheses in call to 'print'

Any suggestions?
Thanks.

Okay. I know the problems. I install python 3.xx rather than python 2.xx.


Hoek version

Version
v1.4.0
Reproduction link
http://www.google.com
Steps to reproduce

What is expected ?
git hub is warning me of a security vulnerability
What is actually happening?
nothing

This bug report is available on Nuxt.js community (#c7017)

Hi. Do npm/yarn upgrade inside your project. Semver/Patches will be applied.

i did that and it was super usefull, but the fixes that were suggested didn't actually patch the vulnerabilities.
npm update fsevents --depth 5,
I ran this mulitple times but it didnt seem to do anything, this was just one of the many updates suggested


CPF stress-test-windows script issues

Device
V560TND
RTE version
--
OSFV version
v560tne-v0.9.1-rc3-testing-fixes
Affected component(s) or functionality
CPF cpu frequency tests under load on Windows
Brief summary
The script is not run automatically, when run manually only uses 4 cores and ends early which is hardcoded
How reproducible
100% on this device and branch
How to reproduce
Run a test like CPF-004.004, check the CPU usage, check che stress-test-windows.ps1 script code
Expected behavior
The cpu should run at ~100% during the whole test
Actual behavior
The clock measurements don't start until the load ends. Then the cpu is running at ~1% usage. Running the script manually does stress the 4 hard-coded cores which results in ~25% usage on this device for about a minute. Even after increasing the hardcoded core count the cpu usage in task manager rises to at most ~60%.
Link to screenshots or logs
$numCores = 4 # <-- Only 4 cores, should not be hardcoded but determined on per-device basis

foreach ($loopnumber in 1..$numCores){
    Start-Job -ScriptBlock{
    $result = 1
        foreach ($number in 1..40000000){   # <-- Ends after 40mln iterations which takes about a minute, the length of the test
            $result = $result * $number           # should not depend on how fast is the DUT
        }
    }
}

Wait-Job *
Clear-Host
Receive-Job *
Remove-Job *

Additional context
Maybe there is some reliable tool we could use instead of writing a stress script
Solutions you've tried
Increasing number of workers and iterations, running the script manually

I have tried to make the jobs time-bound and to get the core count dynamically:
$numCores = (Get-CimInstance Win32_ComputerSystem).NumberOfLogicalProcessors

foreach ($loopnumber in 1..$numCores){
    $end_time = (Get-Date).AddMinutes(60)
    Start-Job -ScriptBlock{
        $result = 7
        while ((Get-Date) -lt ($end_time)){ # Parametrize stress length
            # Is this really the best operation to stress the system?
            $result = $result * $result
        }
    }
}

Wait-Job *
Clear-Host
Receive-Job *
Remove-Job *

This script does not work as well. The core count is ok but the jobs still stop after a couple seconds.

I forgot to mention it when I encountered it a couple weeks ago, but tests under load on Ubuntu were not stable too. After just a couple minutes the clocks were starting to fall to the 200-400MHz ranges and tools like htop were not reporting CPU usage despite specifying that the stress should take exactly 60 minutes in the stress-ng command. This might also be worth looking into


Jenkins connectivity behind a PROXY... Connects to Proxy, but Jenkins Connection timed out.

Hi there,
Is there a way to setup the Docker run to properly setup the machine behind an HTTP Proxy? I'm using the host's network and the http_proxy environment variable. Jenkins automatically identifies that...
$ docker run -d -p 80:8080 --net host -e http_proxy=http://qypprdproxy02.ie.xxxx.net:80 platform.registry.docker.corp.xxxx.net/cicd/jenkins
95d595181a9e1cca8d295804e84ea102baa41056042c99e5a6530af4a59c9285

However, It cannot download the plugins...

Jenkins correctly connects with the Proxy
But it fails to connect to jenkins-ci.org...

 root at pppdc9prd505 in /app/docker-jenkins on git:master o [0:07:11]
$ docker logs -f 95d5
--2015-02-07 08:06:57--  http://updates.jenkins-ci.org/latest//hipchat.hpi
Resolving qypprdproxy02.ie.xxxx.net (qypprdproxy02.ie.xxxx.net)... <IP_ADDRESS>
Connecting to qypprdproxy02.ie.xxxx.net (qypprdproxy02.ie.xxxx.net)|<IP_ADDRESS>|:80... connected.
Proxy request sent, awaiting response... 302 Found
Location: https://updates.jenkins-ci.org/download/plugins/hipchat/0.1.8/hipchat.hpi [following]
--2015-02-07 08:06:57--  https://updates.jenkins-ci.org/download/plugins/hipchat/0.1.8/hipchat.hpi
Resolving updates.jenkins-ci.org (updates.jenkins-ci.org)... <IP_ADDRESS>
Connecting to updates.jenkins-ci.org (updates.jenkins-ci.org)|<IP_ADDRESS>|:443... failed: Connection timed out.
Retrying.

--2015-02-07 08:08:02--  (try: 2)  https://updates.jenkins-ci.org/download/plugins/hipchat/0.1.8/hipchat.hpi
Connecting to updates.jenkins-ci.org (updates.jenkins-ci.org)|<IP_ADDRESS>|:443... failed: Connection timed out.
Retrying.

--2015-02-07 08:09:07--  (try: 3)  https://updates.jenkins-ci.org/download/plugins/hipchat/0.1.8/hipchat.hpi
Connecting to updates.jenkins-ci.org (updates.jenkins-ci.org)|<IP_ADDRESS>|:443... failed: Connection timed out.
Retrying.

Is there anything else I'm missing?
thanks!

proxy should be set in /pluginManager/advanced

@ndeloof can you elaborate how to do that when running this container with a plugins.txt file?

@mllrjb you should be able to run docker build with build args to pass http_proxy to reflect your infrastructure, see https://docs.docker.com/engine/reference/commandline/build/#/set-build-time-variables-build-arg


fix(barrier): Fix bug about barrier collect

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.
What's changed and what's your intention?
PLEASE DO NOT LEAVE THIS EMPTY !!!
After https://github.com/singularity-data/risingwave/pull/4442 , If we use sync(epoch), we will sync all epochs less than or equal to this epoch to S3. Therefore, we need all epochs(le epoch) not to be written after we sync(epoch).
However, if we drop the actors, the small epochs will flow through more actors than the large epochs.
This means that as we sync epoch, other epochs smaller than this epoch will be written.
It may be the reason for https://github.com/singularity-data/risingwave/issues/4774
Checklist

[x] I have written necessary rustdoc comments
[x] I have added necessary unit tests and integration tests
[x] All checks passed in ./risedev check (or alias, ./risedev c)

Documentation
If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.
Types of user-facing changes
Please keep the types that apply to your changes, and remove those that do not apply.

Installation and deployment
Connector (sources & sinks)
SQL commands, functions, and operators
RisingWave cluster configuration changes
Other (please specify in the release note below)

Release note
Please create a release note for your changes. In the release note, focus on the impact on users, and mention the environment or conditions where the impact may occur.
Refer to a related PR or issue link (optional)

However, if we drop the actors, the small epochs will flow through more actors than the large epochs.

Could you clarify a bit for this statement? Why small epochs will flow through more actors than the large epochs?

However, if we drop the actors, the small epochs will flow through more actors than the large epochs.

Could you clarify a bit for this statement? Why small epochs will flow through more actors than the large epochs?

for example, We have actors (1 2 3), and we drop actor 2 3 with barrier1, barrier1 will flow through 1 2 3. But barrier2 will flow through 1. So barrier2 may be collected faster than barrier1.

@xxhZs Let's prioritize merging this PR first since it may fix the recent issues on parallel e2e test failure.

I recall that we also have a similar logic on the meta service for sequentially committing. But that one is for global ordering while this is for compute nodes. Am I correct here?
https://github.com/risingwavelabs/risingwave/blob/3af5280fd76bc2fdcc3cb585bd0d865ef53a5247/src/meta/src/barrier/mod.rs#L780-L819
One concern for me is that we cannot distinguish between the barriers with the state "not collected" and "collected but not able to commit due to ordering" on the await-tree (async stack trace) of compute nodes. 🤔

I recall that we also have a similar logic on the meta service for sequentially committing. But that one is for global ordering while this is for compute nodes, so we need both of them. Am I correct here?
https://github.com/risingwavelabs/risingwave/blob/3af5280fd76bc2fdcc3cb585bd0d865ef53a5247/src/meta/src/barrier/mod.rs#L780-L819
One concern for me is that we cannot distinguish between the barriers with the state "not collected" and "collected but not able to commit due to ordering" on the await-tree (async stack trace) of compute nodes. 🤔

yes, It is. We have similar logic in the meta and computer nodes to guarantee different things.
If the code has not been changed, We cannot distinguish whether the state of barries in ManagedBarrierState's queue is "not collected"  or "collected but not able to commit due to ordering" . But "collected but not able to commit due to ordering" will only appear when we drop actor


App keeps crashing in Android - "COVID Alert keeps stopping" OR "COVID Alert has stopped"

On Android 8.1.0 on Samsung Galaxy, just changed one or more things and went back to the app, the app had crashed with  message "COVID Alert has stopped". Happened every time for me (3 times just now). Used recents/app switcher to switch between apps.
On Android 10 on OnePlus 7, changed font size and display size and tried to switch back to the app and it crashed. Happens every time I try to change a couple of things and go back and then change again and go back (so have to go to settings a couple of times). Just tried 4 times and got this every time. Used recents/app switcher to switch between apps.
Android 8.1.0 on Samsung Galaxy
Android 10 on OnePlus 7

Samsung Galaxy J3 Prime
Android version 7.0
Similar behaviour happened. When try to switch between languages French and English the app has stopped.

@qanitah and @sarasuri I am combining this request with @mmiron's. https://github.com/cds-snc/covid-shield-mobile/issues/607


feat: Add dockerfile for automated builds to dockerhub

What I did
added dockerfile and updated readme to include information about dockerhub
fixes: #
How I did it
added dockerfile
updated readme
How to verify it
we will verify it in the integration
Checklist

[x] Passes all linting checks (pre-commit and CI jobs)
[x] New test cases have been added and are passing
[x] Documentation has been updated
[x] PR title follows Conventional Commit standard (will be automatically included in the changelog)

Should I add docker build to testing for this?

I might have approved too early, I think there is some more to figure out. The main thing, I think, is to not build from pypi but to use the source files. Additionally, I think it is best to have the container function the exact same as the CLI tool itself, so running the container should exit immediately and it should use the simple ape as its ENTRYPOINT. It might take some figuring out. Finally, we should try to use alpine to cut the cruft of the image. This should be able to build fast!

ape as an entry point then the user can't do other things if they need to with the cli within that image, right? Not sure if this is a problem or not.
I wasn't able to successfully get alpine to build unfortunately. There are many dependency errors and issues with pip install.

This article is why I originally decided to go with Ubuntu as a base image.
https://megamorf.gitlab.io/2020/05/06/why-it-s-better-not-to-use-alpine-linux-for-python-projects/

This article details why I'm running into errors trying to build with alpine: https://pythonspeed.com/articles/alpine-docker-python/

This article details why I'm running into errors trying to build with alpine: https://pythonspeed.com/articles/alpine-docker-python/

as per this article I tried to reduce the build size by using python:3.8-slim however then I came into a bunch of errors with unable to execute 'gcc': No such file or directory:
  unable to execute 'gcc': No such file or directory
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for bitarray

  unable to execute 'gcc': No such file or directory
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for lru-dict

  unable to execute 'gcc': No such file or directory
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for pyethash

  unable to execute 'gcc': No such file or directory
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for pysha3

This article is why I originally decided to go with Ubuntu as a base image.
https://megamorf.gitlab.io/2020/05/06/why-it-s-better-not-to-use-alpine-linux-for-python-projects/

Ok, that makes sense!  Ubuntu works.

I might have approved too early, I think there is some more to figure out. The main thing, I think, is to not build from pypi but to use the source files. Additionally, I think it is best to have the container function the exact same as the CLI tool itself, so running the container should exit immediately and it should use the simple ape as its ENTRYPOINT. It might take some figuring out. Finally, we should try to use alpine to cut the cruft of the image. This should be able to build fast!

ape as an entry point then the user can't do other things if they need to with the cli within that image, right? Not sure if this is a problem or not.
I wasn't able to successfully get alpine to build unfortunately. There are many dependency errors and issues with pip install.

Correct, they wouldn't be able to do anything that is not an ape command.
Maybe we don't want that, maybe that is too limiting.. It is just what I see most CLI tools do for their Dockerfiles, so we should at least consider it.

todo:
entrypoints and source code copy
skip for now:
alpine and 3.8 slim

source code copying is working except that installing vyper step is failing.
``$3aa926e91d69 ape plugins list
WARNING: Error loading plugin package 'ape_vyper'
WARNING: $GITHUB_ACCESS_TOKEN not set, skipping 2nd class plugins
Installed plugins:
ape_ens (0.1.0a1)
ape_etherscan (0.1.0a1)
ape_infura (0.1.0a2)
ape_solidity (0.1.0a2)
then when I go to compile docker
```$3aa926e91d69 ape compile
WARNING: Error loading plugin package 'ape_vyper'
WARNING: No compilers detected for the following extensions: .vy```

@sabotagebeats that warning pops up when the plugin isn't currently functioning properly. Try debugging it locally, and then fixing the ape-vyper plugin

this is a preexisting issue: https://github.com/ApeWorX/ape-vyper/issues/22

@sabotagebeats yes, so the solution to that issue will solve this one and allow the plugin to function in the image. However we should take this oppurtunity to think through how the image is built here, and make sure that all plugins are functioning for when we distribute the image.
Perhaps this needs to be "tested" in the image by making sure that ape plugins has no warnings. WIll have to think more on this one.

Perhaps this needs to be "tested" in the image by making sure that ape plugins has no warnings. WIll have to think more on this one.

yes maybe we can do build testing for compilation functionality of the docker image?

It should be pretty straight forward to upgrade the vyper plugin. Let me know if not!
I was going to upgrade all the plugins probably after the next alpha release as the CLI SDK work is wrapping up.

$ docker build .
Sending build context to Docker daemon  16.12MB
Step 1/10 : FROM ubuntu:latest
 ---> 597ce1600cf4
Step 2/10 : RUN apt-get update && apt-get upgrade --yes && apt-get install git python3.8 python3-pip python3-dev --yes
 ---> Using cache
 ---> 676ae34f4e32
Step 3/10 : COPY ./src ./setup.py ./README.md ./pyproject.toml ./setup.cfg ./
 ---> 82edcc2114b7
Step 4/10 : RUN python3 ./setup.py install
 ---> Running in 79fdd6aa6e4a
Traceback (most recent call last):
  File "./setup.py", line 54, in <module>
    setup(
  File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.8/distutils/core.py", line 108, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 447, in __init__
    _Distribution.__init__(self, {
  File "/usr/lib/python3.8/distutils/dist.py", line 292, in __init__
    self.finalize_options()
  File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 740, in finalize_options
    ep.load()(self)
  File "/.eggs/setuptools_scm-6.3.2-py3.8.egg/setuptools_scm/integration.py", line 94, in infer_version
    dist.metadata.version = _get_version(config)
  File "/.eggs/setuptools_scm-6.3.2-py3.8.egg/setuptools_scm/__init__.py", line 177, in _get_version
    parsed_version = _do_parse(config)
  File "/.eggs/setuptools_scm-6.3.2-py3.8.egg/setuptools_scm/__init__.py", line 135, in _do_parse
    raise LookupError(
LookupError: setuptools-scm was unable to detect version for '/'.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj
The command '/bin/sh -c python3 ./setup.py install' returned a non-zero code: 1

filed issue #178

See response (and closure) e.g. just change to VyperToken.vy and it should compile

@sabotagebeats build it locally first, and the move the version.py file over to the image as well, this is generated by setuptools-scm from your git repo during installation.

So I did try to build this locally but I don't have the version.py file. Maybe I'm doing something wrong? It works if I copy the entire directory.

the .git folder is what's needed, which if you don't copy all the files it will install as "dirty" because some of the files are gone
let's stick with copying everything over.can always fix it

You should be able to bind it during the install command, e.g. RUN --mount=source=.git,target=.git,type=bind python3 ./setup.py install

looks like merging main fixed the types issue :)

Is this good to merge and see how docker hub takes it?

looks like merging main fixed the types issue :)

Is this good to merge and see how docker hub takes it?

yes I believe we can merge this and then I can review the integration to make sure it was successful.

this is building properly on dockerhub @fubuloubu @unparalleled-js
It is building immediately and not on version upgrade. Should we build every time a commit is merged, or should we build every time the version is updated?

latest should be building every time
stable and the tagged versions should only build on new version tags
I think this happened today because we did 0.1.1 by accident, and Docker also caught it


[Bug] Can't disable EditingAction "Copy Link with Highlight" on iOS 18.2

Describe the bug
Even with

EPUBNavigatorViewController.Configuration.editingActions set to [], and
navigator(_ navigator: SelectableNavigator, canPerformAction action: EditingAction, for selection: Selection) returning false instantly.

P.S. There is a related comment on Apple Developer Forum — https://developer.apple.com/forums/thread/764462?answerId=818063022#818063022
How to reproduce?

Readium version
3.0.0-beta1
OS version
iOS 18.2
Testing device
iPhone 15 Pro, and in Xcode Preview
Environment
No response
Additional context
No response

Duplicate of https://github.com/readium/swift-toolkit/issues/509


Shared modules/progressive apps support

Currently ScalaJS is not that suitable for mobile development because of monolithic xxx-opt.js.
While loading hundreds Ks of javascript on typing site URL is perfectly appropriate for many situations (B2B, desktop), mobile users won't be such as happy given their slow connections.
Google Closure Compiler allows splitting codebase into multiple modules  via --module and --js arguments:
http://stackoverflow.com/questions/10395810/how-do-i-split-my-javascript-into-modules-using-googles-closure-compiler/10401030#10401030
It could be really great if ScalaJS allow to split generated javascript to multiple modules to avoid waiting for too long for first page to show. For example:

common (scala.* and libraries)
landing page
main page
settings
etc etc etc.

Splitting could be done by packages, by subprojects, by deps etc.

I have the feeling that we already have this idea filed somewhere, but I can't find it. Does someone remember where we discussed this before?

This one: #1787 ?

@scf37 Some part of that discussion was relevant, indeed, but not all of it. The present issue is a neat summary of one particular core feature, so I'll keep it. The other features necessary for #1787 turned out to be already implemented. So I closed #1787 in favor of the present issue + these other implemented features.

One more concern (besides the desired quick loading of the first page) is that using Scala.js commonjs support with tools such as Webpack (which we use currently) is basically unusable during development. When implemented with typical settings (enable source maps, for example), each time (even with a tiny change), our development builds take 40-45 seconds (3-5 second for the scala.js part and the rest for the Webpack part). This is mostly due to Webpack parsing a huge -fastopt.js for every change. We got away, for now,  using the noParse option set for -fastopt.js, but that makes us lose the sourcemaps support. We could try scalajs-bundler, but I highly doubt if it is going to do any better, since at the end of the day, it too runs Webpack.
These tools are optimized for the situations where only few small source files (js, css, etc) change at a time during development. Scala.js in its current form doesn't meet this assumption (it spits out a large "source file").
This feature could the use of promote import/export of many small modules within an app. Then each change should touch only the relevant -fastopt.js file and lower the parsing load on Webpack and other bundler tools.

Agreed. @ramnivas  scalajs-bundler seems like a workaround, not a solution.
Scala.js' is fast despite single large bundle file, nevertheless, it throws the problem to npm/webpack, which have no idea how to deal with the giant, and they won't.
Modules by @JsImport should not be bundled, at first, this is work for webpack alike.
Is it possible to split the *opt.js after it's been emitted, but before webpack intervenes? If yes, how?

I just hit this issue. I need to develop a library on top of many other node.js npm modules and other js libraries and have my library usable by potentially other js libraries (which we can do) as well as other scalajs libraries also 1) deployed through npm and then 2) bundled into clever bundles for web deployment. I think the current output model would replicate the scalajs standard libraries multiple times inside the module. If I could control scalajs output a bit more, without wrapping in the std lib, then I could use webpack and npm to manage all the other assembly parts.

This would probably make it much easier to create a hot code patching tool similar to https://github.com/bhauman/lein-figwheel.

@ubolonton No it wouldn't. You still need to link the entire app at once, and a change in one part of the code can affect the other "modules". This issue is only about the possibility to decompose the one .js output file into several .js files that can be independently loaded (not updated).

This feature should also decrease time of incremental compilation for large projects because only some bundles have to be reassembled.

@scf37 No it probably wouldn't. See my comment just above yours.

@sjrd even for fastOptJS? I thought all names in fast opt mode are deterministic so why not?

Yes, even for fastOptJS. The same comment applies. There are core strategic design decisions in how Scala.js compiles to JavaScript that make this impossible. Deterministic names is not the end of the story. You can look at what the Emitter does if want more precise answers.
Let's not pollute this thread further with random speculation.

I know this issue has been standing still for quite a few months now, but I'm also very interested in this. Not all of my users need all the features of the app, so being able to split the bundles by page would be a huge win

I am starting to look into this, now that 1.0 is mostly done.

From what I understand we need two large building blocks. Definitions of splitting taken from [1]. Comments on feasibility, unmet requirements, misconceptions are very welcome.
Step 1: Split output
Split the SJS output in multiple modules that import each other. This would enable "Entry point splitting": For example, split a user facing app and an admin interface.
Random thoughts:

Splitting by class seems most intuitive
Can bundlers deal with this number of chunks?
What to do with circular dependencies
If we need to be smarter, incrementality might be an issue.
Splitting by artifact (Java style module?) might be another alternative.

Step 2: Dynamic loads
Load modules dynamically import() instead of import so that even single page apps can load partially.
Random thoughts:

Will likely need language support because asynchronicity "leaks".
Probably best driven by the developer anyways (and kept explicit).
Making sure an intended separation stays separated might be a challenge.
If splitting is by artifact / module, introduce "dynamic dependencies"?

Step 1

At least Windows itself has serious issues with large number of files. Splitting every class is going to be completely prohibitive on that platform, regardless of the bundler.
I think we can probably provide a more-or-less custom config to specify where to put the boundaries. The minimum would be to allow to specify a set of packages that should go together.

Dynamic loads

The minimum would be just provide it as is, relying on the developer to use js.import(...) where appropriate.
We could provide some better language support in the form of a js.dynamicConstructorOf[C] and js.dynamicLoad(Obj) where C and O are @JSImport classes and objects. Those could return a better typed Promise than the bare js.import.
We could go further with a js.dynamicLoad { <expr> } that would isolate in <expr> all the references to @JSImport classes and objects, and request them all via an underlying js.import(...) before proceeding with the evaluation of the expr. This should probably be done in a macro outside of the core.

Very glad to this discussion getting restarted!
Split output

This, by itself, will be a huge productivity boost. In our application, for example, we can't use hot-loading with React due to how it treats Context. Generally, context providers sit very high-up in the component tree and subtrees using consumers are updated whenever the provider changes the context. React uses reference equality in detecting if context is changed. With a single -fastOpt.js, with every change, all providers are recreated, and thus creating a new context, thus invalidating all consumers (which, almost always, means effectively every component).

Custom config to define the unit of the split seems fine. Ideally, there are predefined options such as per-class, per-package, etc. Is per-source file a viable option? If so, that may be a fine default, since it mirrors the JS world (and Windows developer choose another option if the number of files becomes a problem).

Dynamic loads
@sjrd By your references to JSImport classes/object, do you mean, dynamic loading will be limited to external javascript-defined code?

By your references to JSImport classes/object, do you mean, dynamic loading will be limited to external javascript-defined code?

Ah! That's a good point. So we'll also need to support that for Scala classes and objects. Somehow ...

Ok, seems like manual config for splitting is a good starting point. Thoughts:

Package prefix is probably the best starting point, no point in specifying scala, scala,collection, etc.
We can simply disallow cyclic dependencies so we only need to detect them, not avoid them.

RE dynamic loads: @ramnivas you beat me to it. I think in practice this will be not as invasive as it seems, we might get away with defining an Async IR node and some compiler primitives. What I'm unsure about is how to make sure the splitting happens as expected by the developer even under complex conditions.

Package prefix is probably the best starting point, no point in specifying scala, scala,collection, etc.

How about a regex over FQCNs?

Great news! This has been the single biggest blocker wrt going all-in on Scala/Scala.js.
Massive :+1:

Thinking further about selectors. Requirements I can think of:

Concept of root module (Scala.js needs this for its runtime)
Fully qualified class name prefix based classification.
Check non-intersection of specifiers without a concrete list of classes (ensures a wrong config is caught early on).
Option to fail if unknown class is present (instead of putting it into a default module)
Option to put unknown classes in a default module.

Notes:
Requirement 3 rules out regexes over FQCNs (/cc @japgolly). Personally I find verifiability more important than flexibility but I'm willing to discuss this.
Requirement 4/5: Unclear what the default should be, especially in view of standard libraries and classes required by the SJS runtime itself (which must be in the root module).
If the default is to have the root module as default module, these things neatly go automatically. However, it exposes the user to the risk of additional classes creeping into the default module (that might not belong there).
Another option would be to have a "smart default" where some "well known" packages are classified into the root module automatically. The downside is that specifying how this default evolves and how one can interact with it (how to tweak parts of it) is more difficult.
Please LMK what you think.

I don't think we need the concept of "root module" per se. Instead I suggest that we systematically put the core JS lib with whatever module contains java.lang.Object (rather than forcing java.lang.Object to be in the "root module"). This will make it simpler from a user's perspective, I believe.

@gzm0 that's a smart consideration, don't worry about my regex suggestion. The scenarios that trade-off would affect are cases where you want to split packages into multiple modules and don't have control over the package structure, like 3rd party libs. In which case, prefixes over regex would be fine so long as:

The prefix is of fqcn, not just package name. In other words, one can specify exact classes.
There's some kind of rule prioritisation mechanism that allows overlap (like acls) so that you could say a.b.C (class) goes to module X, and the remaining a.b. descendants go to module Y. Actually that would probably be pretty useful just for packages too: eg. "java.time goes here, rest of java goes there". You wouldn't want to specify each and every java.xxx where xxx is everything except time. The typical way of doing this is just ordered list of include/exclude rules that allow overlap. Hmmmm maybe req 3 isn't desirable after all...

Thanks for the suggestions.
@sjrd using java.lang.Object as proxy for where the core JS lib goes is excellent. Definitely easier to understand and equally flexible.
@japgolly thanks for the comment. This made me think of another use case: classify certain classes automatically into the "minimal required" module. Probably not something that we'd support in the first version, but it's good to keep it in mind so I design the API to be extensible for that (my current draft isn't).
The prioritization you describe is quite easily done: Say we have two Map[String, String]s: classToModule and packageToModule. For every class, if it's exact name is in classToModule, we take that. Otherwise, we keep looking up it's packages from the deepest one in packageToModule and take the first we find.
This is:

Always unambiguous (i.e. we never get two modules for a given class)
Allows the partial selections you describe (e.g. Map("java" -> "main", "java.time" -> "time")).

So ATM I'm tending towards something like this:
val mappings =
  ModuleMappings.package("java", "main") ++
  ModuleMappings.package("java.time", "time") ++
  ModuleMappings.class("java.time.Foo", "main") ++
  ModuleMappings.default("main")

Note that package and class need to be different because a.b.c can be a package (prefix) and a full class name at the same time (at least in our IR).

@sjrd Other issue: How do I communicate SJS private symbols between modules? If module A needs a thing in module B, how does it access it without polluting module B's "public namespace"? Best solutions I could come up with:

Put everything on single object with a mangled name.
Create "export modules" which contain exported symbols and are always leaves.

I like option 2 more because it is cleaner. Downside is that the config surface becomes more complicated.

I had option 2 in mind as well. I think it's the only bulletproof thing to do.

Yeah... Downside is that with this, having a single file is not a mere special case of the multi file output :-/

I have been thinking about how to build the interface to this for a while and I'm not sure the proposed mappings are really what we want.
Take the following config:
val mappings =
  ModuleMappings.package("myapp.core", "core") ++
  ModuleMappings.package("myapp.frontend", "frontend") ++
  ModuleMappings.package("myapp.backend", "backend") ++
  ModuleMappings.default("core")

There is (or might be) an implicit assumption in this configuration that frontend and backend depend on core but not on each other. Since this is implicit, it cannot be verified.
Therefore, if by happenstance, something in backend refers to frontend, the resulting module structure will be valid, but broken.
Hence, I'm toying with the idea of requiring this to be explicit. Something like:
val modules = Seq(
  Module("core")
    .withMappings(Mappings.default()),
  Module("frontend")
    .withMappings(Mappings.package("myapp.frontend"))
    .withDeps("core"),
  Module("backend")
    .withMappings(Mappings.package("myapp.backend"))
    .withDeps("core"),
)

This would both avoid this problem and make some of the implementation easier.
A potential downside is verbosity in configuration. A further caveat is how to deal with transitive dependencies (if we allow them, we can make a similar argument about undetected dependency creep, although it would only manifest later).
Opinions?

I like it, looks similar to SBT's aggregate and dependsOn.
To avoid stringly typos would we be able to do something like:
val core = Module("core").withMappings(Mappings.default())
val modules = Seq(
  core,
  Module("frontend")
    .withMappings(Mappings.package("myapp.frontend"))
    .withDeps(core),
  Module("backend")
    .withMappings(Mappings.package("myapp.backend"))
    .withDeps(core),
)

?

#3965 contains an RFC for an interface specification.

I am not so sure that this package-based modules idea is the best way to proceed. A few reasons:

Package may not be the right level of separation. For example, we have a package paya.ui that has a bunch of generic UI components (Button, Slider, Dialog, etc.) defined. Not all parts of the code need not all the components.
It will make it hard to deal with some of the expected functionality in the JS world. For example, as I mentioned in https://github.com/scala-js/scala-js/issues/2681#issuecomment-559888858, hot-loading in the presence of React Context with less granular JS files will not work well.
It is significantly different than the JS way, requiring a different way of thinking, resulting in a higher barrier for newcomers considering ScalaJS. Working and keeping up-to-date with Webpack will require special ongoing effort, which our relatively small community may not be able to keep up with. And then there are new tools that exists or are being proposed in the JS world (for example, rollup, parcel, ESBuild, SWC). We cannot possibly support each of these tools.

So what could be done? Please note that I am sure that I have gotten a few details off. But hopefully, this will serve as a basis for discussion. Also, I may be repeating a few things said by me/others in the past, but trying to put a more complete picture here.
The core idea is to mimic the JS world as closely as possible. Then Webpack et al will have their assumption met (specifically that only a few files change at a time during development). I also don’t think we will need scalajs-bundler if this is implemented. Letting Webpack parse only a few JS files will be very performant and will not need the LibraryMode workaround and then there is very little is left for scalajs-bundler to do.
Consider the following set of classes:
class A {
  def display(): String = {
    val b = new B()
    b.hello()
  }
}

class B {
  def hello(): String = {
    "hello"
  }
}

Should lead to the creation of the following files:
// A.js
import {B} from './B';

export class A {
  display() {
    let b = new B()
    return b.hello();
  }
}

// B.js
export class B {
  hello() {
    return "hello"
  }
}

Basically, every type gets its own dedicated js file and includes an import for each referred type. One obvious objection is the sheer number of files produced. But a quick check in our real project reveals that we have 35770 js files (most of those coming from node_modules) and that doesn’t seem to be a problem (true, I am on a Mac; but Windows users, too, presumably have similar number of js files in their projects). Also cc @Shadaj who has an idea of dealing with this issue, should it become a problem.
This, by itself, solves a bunch of problems. For example, unless the B scala type is modified, B.js will remain the same (at least in the fastOptJS mode, I hope). This means the React Context problem (which relies on reference equality) will no longer be an issue. Webpack will have minimal work to do during iterative development. And scalajs-bundler will be no longer required (short of providing integrated experience from within SBT; but that is, IMO, not super critical).
Now onto dynamic imports. Consider the following modified version of A.
class AA {
  def display(): Future[String] = {
    dynamicImport {
      val b = new B()
      b.hello()
    }
  }
}

Assuming that we have defined dynamicImport as the following (in Scala.js alongside JSImport and such):
def dynamicImport[T](thunk: => T): Future[T] // a marker method

Will lead to the creation of the following A.js (B.js unchanged):
export class AA {
  display() {
    return import('./B').then(({B}) => {
      let b = new B()
      return b.hello();
    });
  }
}

Basically, each type referred inside a dynamicImport block will result in a corresponding import statement. With this generated code, Webpack will create a separate chunk for the code inside the import block.
Let’s also consider a case where static and dynamic imports share a type.
class AAA {
  def display(): Future[String] = {
    dynamicImport {
      val b = new B()
      b.hello()
    }
  }

  def display2(): String = {
    val b = new B()
    b.hello()
  }
}

Here, we could do nothing special and import B statically and then again dynamically and leave it up to Webpack to understand (which correctly handles the situation by not creating a separate chunk to load B). Or could do something special to avoid dynamically importing types that are already required in the static scope.

@ramnivas thank you for raising your concerns!
I have a couple of clarification questions:

Package may not be the right level of separation. [...] Not all parts of the code need not all the components.

What I read between the lines is that you do not want to define a module per class (i.e. UI component) yourself here? (which with the current proposal is possible, but maybe cumbersome indeed).

It will make it hard to deal with some of the expected functionality in the JS world. [...] hot-loading in the presence of React Context with less granular JS files will not work well.

Could you clarify how the context maps to files and reloads? I do not know how react contexts are typically set-up and how the hot-loading works. From what I have understood so far, wouldn't it be enough to simply put the context provider in a separate module?

It is significantly different than the JS way, requiring a different way of thinking, resulting in a higher barrier for newcomers considering ScalaJS.

Could you clarify what the JS way is and how what we are doing is more different than necessary?

RE your proposal of per-class splits

How does this deal with cyclical dependencies? E.g.
class A { def foo = new B }
class B { def foo = new A }

I have a feeling that having cyclical imports is a recipe for disaster.

@sjrd do you have numbers on how many files are too much for a windows system? Just so we actually have ballpark numbers to compare against each other.

Not really. I just remember that 1 per class was too much; it was unusably slow.

What I read between the lines is that you do not want to define a module per class (i.e. UI component) yourself here? (which with the current proposal is possible, but maybe cumbersome indeed).

Indeed. In a way, the real difference between the current proposal and what I desire is no separate dependency declarations; referring to a type normally should suffice.

Could you clarify how the context maps to files and reloads? I do not know how react contexts are typically set-up and how the hot-loading works. From what I have understood so far, wouldn't it be enough to simply put the context provider in a separate module?

Indeed, it would be sufficient to put the context provider in a separate module. Basically, as long as that module remains unchanged on the filesystem (thus not forcing the hot-loading mechanism to load that file), everything should work fine. The issue is that the provider and consumers often are a part of the same package (it feels natural that way). For example, in our case, for the composition editor, here is (an abbreviated and simplified) component hierarchy.
CompositionEditContext // has a state "currentComposition"
  Context Provider // providing context around "currentComposition"
     CompositionEditor
         SectionsEditor
             SectionEditor[] // Use the context and effectively update "currentComposition"
                 LineEditor[] // ... same
                    ClusterEditor[] // ...same

The lower level components access the context provided up in the hierarchy (using React.useContext()). The way React works is if the currentComposition changes (in reference equality way), it will rerender all consumers of that context.
When I modifying, say, ClusterEditor, I want hot-loading to only load that code. However, if I put the provider in the same module as consumers, then modifying ClusterEditor will end up creating a referentially-unequal context and thus recreating the whole hierarchy.
We have many such prodivers (Selection, Environment, Mode, etc) and they all are modelling a part of composition editor functionality. Putting them in a separate package will require explicit thinking about how Scala.js works and that will be a deviation from the way JS developers think.

Could you clarify what the JS way is and how what we are doing is more different than necessary?

In the JS world, module dependency is a local concern: I need a module (often consisting of just a class or a handful of functions), I declare thus in the dependent module and that is it. The current proposal requires me to think about the system as a whole and that means for a newcomer to the team, there is one more thing to learn.

How does this deal with cyclical dependencies?

JS seems to allow this. For example, it allows:
// A.js
import {B} from './B';

export class A {
  display() {
    let b = new B()
    return b.hello();
  }

  message() {
    return "A's message"
  }
}

// B.js
import {A} from './A'

export class B {
  hello() {
    return new A().message()
  }
}

So we may not have to do anything specific i.e. add imports for referred types and leave it to Webpack et al to deal with it.

@gzm0 Thanks for asking questions that gives me an opportunity to express the application developers' point of view.

Not really. I just remember that 1 per class was too much; it was unusably slow.

From application developer's point of view, 1 js file per 1 scala source file would suffice. More specifically, there need not be a separate js file for each nested types/companion objects, for example. Looking at our system, for example, for AlankarRouting.scala, we see the following class files:
AlankarRouting$.class
AlankarRouting$AlankarRoute$.class
AlankarRouting$AlankarRoutingParams$.class
AlankarRouting$AlankarRoutingParams.class
AlankarRouting$Default$.class
AlankarRouting$View$$anonfun$$lessinit$greater$1.class
AlankarRouting$View$.class
AlankarRouting.class

It would suffice if all these led to a single AlankarRouting.js.
Looking at our app, we have .class/.scala ratio of about 5. So that would be some saving without any impact of developers.

The current proposal requires me to think about the system as a whole and that means for a newcomer to the team, there is one more thing to learn.

It's important to note that this point of view is from the perspective of a JS programmer working on a Scala.JS project. To any Scala developer, a few SBT settings barely register on the learning curve, where as learning webpack and integrating it with SBT is a huge amount of learning, time and effort.
I'd be very happy if this solution were able to output exactly what tools like webpack, rollup, etc  need so that workflows like what @ramnivas suggests are possible. The part I don't agree with is that workflow being mandatory, or a JS "build tool" being mandatory in the Scala toolchain. I'd hope that a solution could accommodate both, so that a Scala dev can enter a few SBT settings and out pop some modularised JS files that can be used immediately. From a Scala devs point of view, that's the path that requires the least learning, and the path that most easily integrates into the Scala dev workflow.

a few SBT settings barely register on the learning curve

Maybe I misunderstand your point, but having worked with both sbt and webpack a lot, I think you're overestimating the usability of sbt for an average Scala developer and underestimating the amount of sbt settings needed to replicate a useful subset of webpack in sbt. The innate complexity of bundling functionality isn't going away just because it was translated into sbt.
Bundling Javascript apps is a solved problem in Javascript. Regardless of what I think of sbt or webpack, Scala.js is just too small of a world to be developing its own Javascript bundling tooling. Well, any more than is strictly necessary for effective interop with existing JS bundlers.
When I have an issue with webpack I can google it and easily find a quality answer because that's what millions of people in the world use for bundling. When I have an issue with scalajs-bundler, well, good luck to me trying to understand it, let alone solve it. It'll be gitter spelunking all day at best.

I've worked with lots of Scala devs (including some fresh converts with no prior experience) and I've never met one who couldn't copy a few lines of SBT config into their build. I don't know if @gzm0's  original idea above is the best way to go, seems good enough to me, but I'm positive that what was presented above could be understood by the average Scala team/dev with very minimal effort.
webpack is the complete opposite. I also have a lot of experience in it (unfortunately) and I've consistently had a very different experience that what you describe. "You can google it and find a quality answer"? What?? First of all there are so many answers all over the internet because there are so many questions because it's so confusing, unstable and flakey. Secondly the vast majority of answers don't work, nor does the official documentation of webpack itself nor the many, many webpack plugins' official documentation because webpack changes so often and so fast and breaks everything in its path, all the damn time. I've wasted so much of my life literally copying-and-pasting settings from the official webpack homepage that don't work anymore because they don't update their documentation enough. Supporting users who want to use webpack: great, go for it. Requiring that Scala developers learn, intergrate and use webpack is a terrible idea, and I think you only believe it's feasible because as you say yourself: you've worked a lot with it. A Scala dev who's never used it isn't going to feel that way.
And yes, comparing it to scalajs-bundler makes it look favourable but I don't recommend people use scalajs-bundler; that's not the alternative I'm proposing (although if it I were I would 100% agree with you).
No, what I'm proposing and apologies if I wasn't very clear, is that we should think of the Scala dev perspective, seeing as this is Scala.JS. Scala devs shouldn't have to learn JS tools (nor learn how to integrate them with their Scala tools). Scala.JS already generates usable JS directly from SBT (via {fast,full}OptJS). We're talking about adding a feature to Scala.JS in this ticket. Using this feature once implemented, shouldn't break the existing ability to generate usable JS directly from SBT. Whatever downstream effects or tradeoffs we make whilst maintaining that property is something we should discuss and consider, but the idea of leaving that core property behind is the one aspect I feel very strongly that we should not do.

@japgolly Generating bundles is one part of the equation and not that easy in any case. In our app, I see about 50 "modules"--small and big--that is not needed by most users. Going the SBT way would mean hand-crafting those many modules and declaring dependencies for them.
Assuming that hurdle is somehow crossed, we need to make sure that the right modules are loaded before the corresponding functionality is handled. Who/how should this be done?
Let's say (a real use case from our app), where the "print composition" is a rarely used functionality and has considerable code for layout manager and such. I imagine with my proposal, I will write the following code:
def clickPrint() = {
  dymanicImport {
     PrintSupport.print(...)
  }
}

And I am done. Webpack will take care of creating a separate chunk for print-related functionality and make sure that it is loaded before "PrintSupport.print(...)".
How do you imagine managing the same through just SBT?
I am no fan of Webpack. For a long time (and to some extent still) don't understand what/how it does many things. But it works. I suspect that if one were to create an alternative tool, that will have to essentially duplicate what Webpack does.

Scala.JS already generates usable JS directly from SBT (via {fast,full}OptJS)....Using this feature once implemented, shouldn't break the existing ability to generate usable JS directly from SBT.

I imagine a mode setting that dumps the code into one js file or into many js files.

@japgolly Actually, if this feature is implemented in a JS-like way (my proposal), then you don't need any bundler at all. Most browsers now natively support modules https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import (IE being, as usual, an exception).
So those who want to use Webpack can do so (likely for reasons such as CSS/SCSS processing, image minifying during build, and to reduce the number of separate chunks to be downloaded) and those who don't like Webpack etc. can use browser's native module support. With HTTP2/HTTP3, downloading many small files is less of an issue, so this is where things are heading, anyway.

Thank you all for your comments. This is very insightful.
What I'm hearing is the following:
@ramnivas would like the system to be as automated as possible (high automation, low control), forwarding everything to webpack where feasible. A use-case he is particularly interested in is hot-reloading for React.
@japgolly and @raquo both want a simple way to generate N > 1 output modules. There is some disagreement on which tool is easier to use / which community is stronger / has better support.
Please LMK if this summarizes the discussion appropriately.

With regards to circular dependencies: If I amend @ramnivas's example to:
export class B extends A {
  /* snip */
}

The example fails depending on the order of which module is first loaded. Since this can perfectly happen in Scala code, Scala.js would be forced to write multiple modules per class and move away form ES2015 class declarations and instead set members mutably. I'm afraid this is unacceptable.
This would leave us with the problem of finding a maximal acyclic graph partition of the cyclic class dependency graph. I'll do some research on the feasibility of this.

Stackoverflow answer with algorithms for finding cycles: https://stackoverflow.com/a/2794683/1149944

Seems what we want is strongly connected components: https://en.wikipedia.org/wiki/Tarjan's_strongly_connected_components_algorithm

Can't you guys start with subprojects? Must be simpler, no recursive deps by definition, also looks like good practice for modular javascript applications,

@ramnivas would like the system to be as automated as possible (high automation, low control), forwarding everything to webpack where feasible. A use-case he is particularly interested in is hot-reloading for React.
Please LMK if this summarizes the discussion appropriately.

Pretty much. I will clarify that by webpack, I really mean "webpack" 😄 i.e. any bundler system that exists or will exists in the future (including directly using a modern browser that supports modules). The main use case for me is the ability to defer loading of many many (in the range of ~50) modules, which would be painful without a JS-like approach.

With regards to circular dependencies: If I amend @ramnivas's example to:
export class B extends A {
  /* snip */
}

It seems that 'A' will be always loaded before B. Either A is loaded because some other class has a dependency on it or it be loaded in the process of loading B.

@sjrd I'm writing the code to import SJS definitions from one module into another. For this, we have two higher level options:

Import a module wildcard and select on that module (you have expressed that you prefer another solution).
Import the relevant identifiers individually / directly (e.g. import $c_frobber from 'module.js').

For the second solution we need to know what to import. We can do that based on:

What each imported module defines. Cons: We over-import and over-invalidate.
What each importing module needs.

The second option seems clearly desirable here.
For this in turn, we have two options:

Record as we build the trees in Emitter (on WithGlobals).
Pass it down as analysis.

If I remember correctly, you have expressed preference for the second solution. However, it seems to introduce substantial coupling between the Analyzer and the Emitter. As such, I still feel a solution recording the required imports as we build the trees is easier to build and maintain. WDYT?

I thought we could simply import what a module defines. We can determine that just from the fields of the target LinkedClass, which means we don't need to record anything nor consult a full Analysis. Of course that means we can over-import, but only of members that are anyway defined by a module that we have to refer to. So it won't result in importing too many modules, only too many fields of the modules. IMO that seems reasonable for now, and we can be more precise in the future as an optimization.
If you think we should only import the fields we're going to use, then recording as we build the trees seems preferable, IMO.

So it won't result in importing too many modules, only too many fields of the modules.

Ah, yes indeed. This is not the main concern I have with over-importing. It's that we might be forced to re-emit a module even though it did not change (but one of its dependencies changed). This would not support the use cases for fine grained modules (which is what I'm prototyping for right now, to find out whether "too slow due to too many files" is a problem and if yes how big of a problem).

True. That could potentially cause more re-generation than necessary. Although I would assume that the set of variables exported for any given LinkedClass wouldn't change often.

So now linking a PR closes the linked issue...

Forgive my delayed response here. Ok so @ramnivas has pointed out that modules can be used directly in browsers. It's a bit of a psychological barrier for me to overcome but MDN is saying Chrome/FF have supported dynamic imports since 63/67 so I guess this is the new world! In which case, I think my earlier points are less valid now. Looking forward to seeing how this turns out!

More context/discussion on this in #4010.

Cross posting from #4111 (v2 prototype). We are discussing about a solution where each @JSTopLevelExport would have to specify its target module (with a default). As such, there would be very little linker configuration:

Module initializers (as right now, but per module).
ModuleSplitStyle (SmallestModules, FewestModules)

Would this be a viable alternative for everybody's use cases?

For my personal use cases, I'm fine with that.
I would like to just confirm though: Let's say I have two top level SBT modules A & B with a @JSExportTopLevel in each, and a shared SBT modules C & D that they both A & B on. I'd like to end up with three JS modules: A, B, and a shared module of C+D (roughly speaking). Specifying the target modules in A & B is no problem but how would one go about extracting the parts of C & D that are shared by both A & B? I sense webpack in the answer but what are the SJS outputs / webpack inputs going to look like?

For your example, you'd get 3 files (with FewestModule splitting mode):

a.js: everything that is only needed by A, imports root.js
b.js: everything that is only needed by B, imports root.js
root.js: everything that is needed by A and B and Scala.js itself.

With SmallestModule splitting mode, you'd get:

a.js: the top level export only
b.js: the top level export only
tons of small files with minimal imports between them (with class level granularity).

Both of these you can but don't need to run them through webpack (but the second one is probably better suited for this).

One distinction is that @japgolly was talking about a case where A and B are different sbt project. To achieve the result shown by @gzm0, they would either have to both be put in the same sbt project, or have an additional sbt project E that depends on A and B and contains no code. You'd then use E/fastOptJS to produce all the .js files.

Ok cool. Yes, as @sjrd says I was mostly concerned about different sbt modules/projects. If I have to create an exports-only sbt project E, that's actually no problem, especially if there's a FewestModule mode that avoids me needing to involve webpack, well that's a perfectly acceptable trade-off and I'd be very happy with that. Thanks!!

This looks good for our use case. We are likely to have only one JSExportTopLevel and SmallestModules should suffice our needs.


Switch cropperWindowBuffer to const

Taking a stab at https://github.com/wulkano/kap/issues/25. I noticed in the renderer index.js the cropperBounds width and height are being subtracted by 4px. Seen [here]:(https://github.com/wulkano/kap/blob/master/app/src/renderer/js/index.js#L100-L101)
cropperBounds.width -= 4;
cropperBounds.height -= 4;

So I added a cropperWindowBuffer which sets the cropperWindow preview 4px wider and taller, but correctly exports the same dimensions now.

Amazing again, @stursby! You just fixed two issues with one PR 😌 – #24 and the fact that the recorded area was not the same as the one shown in the marquee ✨ Thanks!