Error deploying in OpenShift because of chown

What happened?
Hi, I'm trying to setup a POC of Reposilite for my organization, it runs perfectly via podman on my development machine,  but when I try to deploy it in an openshift cluster the pod crashes right after the start. Below you will find the log entries before the crash, and here's my deployment yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dep-reposilite
spec:
  replicas: 1
  selector:
    matchLabels:
      deployment: reposilite
  template:
    metadata:
      labels:
        deployment: reposilite
    spec:
      containers:
      - name: reposilite
        image: dzikoysk/reposilite:3.2.0
        ports:
        - containerPort: 8080
          name: tcp-8080
          protocol: TCP
        resources:
          limits:
            cpu: 500m
            memory: 256M

Does the container need root priviliges to run? In this case this would break one of my company's security policy.
Thanks for the support!
Reposilite version
3.x
Relevant log output
chown: changing ownership of '/app/data/lost+found': Operation not permitted
chown: changing ownership of '/app/data': Operation not permitted
chown: changing ownership of '/app/entrypoint.sh': Operation not permitted
chown: changing ownership of '/app/reposilite.jar': Operation not permitted
chown: changing ownership of '/app': Operation not permitted
chown: changing ownership of '/var/log/reposilite': Operation not permitted
runuser: may not be used by non-root users

Hey :) Root is needed, so container can create dedicated user for Reposilite and run application as non root user:

https://github.com/dzikoysk/reposilite/blob/main/entrypoint.sh

By default application would run as root, so this is exactly needed to launch application as non root user.

Thanks for the quick reply :) Maybe I'm missing something, but what's the need for a dedicated user? I mean, as a java application it should be sufficient for the container to run command "java -jar ...". I don't have experience with kubernetes, but in Openshift containers run as a non-root unique user, separated from other system users. Due to this, I'm afraid I won't be able to try Reposilite in my environment.
Thanks :)

It's a good practice to not run application inside container as root user and usually such software creates dedicated user, see:

https://github.com/dzikoysk/reposilite/issues/1200
https://github.com/dzikoysk/reposilite/issues/1634

Speaking of the issue - you don't need root to run Docker container, but inside of the container it should run as a root to bootstrap permission system. In general that's how Docker containers work out of the box, I don't have any experience with OpenShift, so I'm not sure how it works under the hood. One thing I see in your logs is that build script is unable to chown workdir directories:
https://github.com/dzikoysk/reposilite/blob/68e508f35c7172bdd94c1471934103d9ab183698/entrypoint.sh#L9-L10
Results in:
chown: changing ownership of '/app/data/lost+found': Operation not permitted
chown: changing ownership of '/app/data': Operation not permitted
chown: changing ownership of '/app/entrypoint.sh': Operation not permitted
chown: changing ownership of '/app/reposilite.jar': Operation not permitted
chown: changing ownership of '/app': Operation not permitted
chown: changing ownership of '/var/log/reposilite': Operation not permitted

Did you mount fs/part of fs to that container as a volume?

Openshift works like Kubernetes and they are different from standalone Docker.
I tried both mounting a PersistentVolumeClaim as the "/app/data" path (this is something pretty usual in kubernetes-like environment to ensure data persistency) and running without an external volume just as a test, results are the same. I think the problem stays in the chown and runuser commands, as they cannot run without root privileges. Openshift documentation (https://docs.openshift.com/container-platform/4.8/openshift_images/create-images.html) says:

By default, OpenShift Container Platform runs containers using an arbitrarily assigned user ID. This provides additional security against processes escaping the container due to a container engine vulnerability and thereby achieving escalated permissions on the host node.
For an image to support running as an arbitrary user, directories and files that are written to by processes in the image must be owned by the root group and be read/writable by that group. Files to be executed must also have group execute permissions.
Adding the following to your Dockerfile sets the directory and file permissions to allow users in the root group to access them in the built image:
RUN chgrp -R 0 /some/directory && \
   chmod -R g=u /some/directory

I don't have experience in building images (but I'm gonna try), but it's the first time I have this kind of issues running a container inside Openshift.

Let me know if you'll find something related to this issue :)

I've done it :) !
I've built a new image, starting from official reposilite:3.2.0  and fixing folder permissions. Here's my Dockerfile (it't the alpha version, next I'm gonna add a full entrypoint.sh supporting the optional parameters as in the original version).
FROM docker.io/dzikoysk/reposilite:3.2.0

WORKDIR /app

RUN chgrp -R 0 /app && \
    chmod -R g=u /app

ENTRYPOINT ["java", "-jar", "reposilite.jar"]

Now Openshift deploy works and reposilite it's up and running.

Nice :)
I think we can close this for now, as this is still just the first issue with OpenShift. If more people will request support for OpenShift we can go back to this case and reconsider official support.

@dzikoysk would you consider adding this to the default docker image. Happy to submit a PR.
RUN chgrp -R 0 /app && \
    chmod -R g=u /app

We run this inside openshift devspaces and would like to not maintain our own version of the container image. Seems the only thing is missing is allowing group ownership of app folder

@antoniordz96 hmm... I think it shouldn't be a breaking change, because we expect that the entrypoint.sh file is executed by root user anyway. What I'm not entirely sure is why it actually helps in your case, I'd assume that the app directory should be already accessible to root :thinking:


update on-demand build

Follow on to #3634 . Fixes one bug in it, and restricts to running only on master branch.

It would be nice to have a commit message.

I thought I did. I guess not. I updated the commit message.

I thought I did. I guess not. I updated the commit message.

Thanks!


Does locator.click() use scrapy custom headers?

I'm using a scrapy middleware 'RotateAgentMiddleware' to set request headers, which seems to work fine in most cases.
However, I'm noticing that when I use the .click() method on a locator (e.g. page.locator('div.something').click()) inside of my parse method, it appears that these custom middleware headers are not being used.  I determined this by debugging with "headless: False", setting a breakpoint immediately before the click(), then using chrome dev tools in the visible browser to see the request headers.
Is that correct (that the scrapy headers are not used in the .click())?  If so, is there a way I can send my custom headers with any .click() method?
Thank you!

That is correct, any network operations resulting from manipulating the page once you're in the callback are processed directly by Playwright and bypasses the Scrapy workflow[1]. This is briefly explained here for PageMethod objects, the same applies to this case. Thanks for noticing it, I will add it to the README.
Regarding modifying headers, see the docs for the PLAYWRIGHT_PROCESS_REQUEST_HEADERS setting.
[1] they can still be intercepted by the PLAYWRIGHT_PROCESS_REQUEST_HEADERS and PLAYWRIGHT_ABORT_REQUEST settings

Ok, that explains it.
The documentation for PLAYWRIGHT_PROCESS_REQUEST_HEADERS seems a little confusing to me.  It appears that I can use that to set the same headers I'm using with Scrapy, which I assume would persist for any page methods?  e.g. PLAYWRIGHT_PROCESS_REQUEST_HEADERS = custom_headers where I copy all the header values I'm using.
But, the part where it says setting PLAYWRIGHT_PROCESS_REQUEST_HEADERS=None says only headers set by Playwright will be sent.
Is that not what is happening in the first part of the documentation where I'm setting my own custom headers?  What is the difference?
Thank you!

scrapy-playwright uses Route.continue_ internally to modify requests, and PLAYWRIGHT_PROCESS_REQUEST_HEADERS controls whether or not and how the optional headers argument is passed.
There are 3 possibilities for PLAYWRIGHT_PROCESS_REQUEST_HEADERS:

scrapy_playwright.headers.use_scrapy_headers, the default value, code available here. As explained in the docstring, this populates the headers with whatever is set by Scrapy (e.g. my middlewares, both built-in and user-defined) but only for navigation requests.
None. Do not override headers, let Playwright handle them.
A user-defined function. For instance, if you wanted to set your own headers for all requests, both navigation and background.

Got it, this is clearer now.
For 3, is the user-defined function called once or multiple times inside each scrapy.Request?  i.e. are the headers set for the entire Request?
For example, if I'm using a random function to select a header from a list of headers in the user-defined function, could that be changing values multiple times inside of a given scrapy.Request?

For each Scrapy request there is at least one Playwright request (for the main URL), but it's possible that more Playwright requests are generated to retrieve additional assets (images, style sheets, etc). The function will be called for all of those generated Playwright requests. I've tried to clarify the behavior on a816f86046e967679bf9ae3faeece448dcefb53e.

I think that is what I'm seeing in my logging, that the custom_headers function is being called many times for a single scrapy.Request.  So, the random function is randomizing every time.
Ideally, what I would want is to set the custom_headers once for the entire scrapy.Request, including the mail URL and any following requests made in Playwright.  Although, it isn't obvious to me how to do that, other than with constant values as in the example shown.  Or, is there a way to do this?

One idea that comes to mind is setting a header in the Scrapy request (e.g. when producing the request in the spider or in a middleware) and then picking up that header in the custom
class MySpider(scrapy.Spider):
    def start_requests(self):
        yield scrapy.Request("https://example.org", headers={"asdf": "qwerty"})

async def custom_headers(
    browser_type: str,
    playwright_request: PlaywrightRequest,
    scrapy_headers: Headers,
) -> dict:
    scrapy_headers_str = scrapy_headers.to_unicode_dict()
    playwright_headers = await playwright_request.all_headers()
    playwright_headers["asdf"] = scrapy_headers_str["asdf"]
    return playwright_headers

Thanks, that works!


fix: change build dir to dist

I'm not sure whether this is intentional or not, but I think this is just a small oversight?
The docs say the addons are in the dist directory, but they're not - and the build script builds them into the build directory...

@imsnif this is because we have not issues the 2.2.0 release yet, which will happen within the next minutes.
Sorry about the trouble.

@imsnif this should be fixed in the 2.2.1 tag that just got pushed.

Cool, thanks! Sorry - it just stopped working for me with the latest pull.


Loadbalancer external listener for the kafka service

Hi,
I seen there is option to create nodeport service, Is there any option to create LB service for the kafka pods.
I want to allow external access for the kafka broker. How to set the external advertisment address for kafka broker.

Any one can help on this plz ?

You might have better luck with Strimzi if you want to have LoadBalancer or Ingress resources.
Direct links to docs - https://strimzi.io/docs/operators/latest/overview.html#configuration-points-broker_str

any possibility of this being achieved with the current helms ?? … Any help will be much appreciated …

Yes, these charts can be used, but they don't currently offer such a feature, and are no longer supported, so PR to add such a feature might not be approved.
The Confluent operator is where Kubernetes features are being added. https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/networking/external-access-load-balancer-deploy
But this will require a license after 30 days


Payload cannot receive though lamda invoke call.

This is a Bug Report
Description
For bug reports:
call the example project by
curl http://localhost:5000/bff

and I try to print out the the event from bff.backend
const backend = (event, context, callback) => {
  // info.log(event, context)
  console.log('Recieved event', event)
  callback(null, { message: 'success' })
}

The event is empty.
Actually i expected the event will have the payload. like
{
      foo: 'bar'
}

Additional Data
Node Version:       7.6.0
Serverless Version: 1.8.0
Mac 10.12.3

Very good catch @yoyeung. This was a weird one. I've got a fix coming soon.


Adding a Grafana monitoring chart with GKE nodes types (reserved, spot, on-demand)

Adding a Grafana monitoring chart with GKE nodes types (reserved, spot, on-demand)

/gcbrun

/gcbrun

/gcbrun


Epic fail at the first step, doing 'Explore Azure SQL Database' (dp900-01-sql-lab.md)

Summary
Raw Error
ERROR TYPE
Provisioning is restricted in this region. Please choose a different region. For exceptions to this rule please open a support request with Issue type of 'Service and subscription limits'. See https://docs.microsoft.com/en-us/azure/sql-database/quota-increase-request for more details. (Code: ProvisioningDisabled)

Remove this guide if students don't have a chance to complete it

@afelix-95 will review this

Some regions have restricted access for creating new Azure SQL servers and databases. For example, if you choose East US as the region for your server it might not let you deploy the resource, so you need to change the location of your server to another option. For me, West Europe worked, but you can try any other option.


Have the AppManager set up routing for AppServers

https://ocd.appscale.com:8080/job/Daily Build/1043/

Precise: https://ocd.appscale.com:8080/job/Daily Build/1051/

Precise: https://ocd.appscale.com:8080/job/Daily Build/1053/
Trusty: https://ocd.appscale.com:8080/job/Daily Build/1054/

:+1:

Precise: https://ocd.appscale.com:8080/job/Daily Build/1063/
Trusty: https://ocd.appscale.com:8080/job/Daily Build/1067/


Add Caffeine as alternative local cache implementation

Done

Very cool! 😸

@ben-manes
Thanks for great lib!


BUG: Hover effect not visible on active link in navigation bar #156

The hover effect on the active link in the navigation bar is not visible. Specifically, when the "Home" link is active, the hover effect does not apply, making it difficult for users to identify the active link visually.
Now I have fixed as you can see in the below video
https://github.com/Suchitra-Sahoo/AgriLearnNetwork/assets/109781385/f3d3424a-dc37-410d-a0b9-6c291e19519e

issue #156

@Suchitra-Sahoo I think it should be at least Level 1. According to the GSSOC team, they said to give Level 1 to documentation. At least I have contributed something to the code.


Examples gallery broken (again)

In re: #1822, the sidebar seems to have been added again.  It's still hideous, and it's still breaking the Examples gallery.
Let's 86 it once more!

I fixed this in 0953d5c, https://github.com/mbostock/d3/wiki/Gallery/_history. I'll leave this for someone else to verify (and close). The fixes should continue to work if/when the sidebar is removed again.
The original issue is in Firefox, so here are the before afters in Firefox & Chrome for comparison:
Firefox – before

Firefox – after (fixed)

Chrome – before

Chrome – after

Someone inexplicably reverted your fix, so I just made another set of fixes myself. Thanks for the report.


Change the default setting of the some feature flags to true.

Summary
Change the default setting of the some feature flags to true.

upload-file-to-prompt-node
web-search-node
choose-model
anthropic

Related Issue

https://github.com/giselles-ai/giselle/issues/44
https://github.com/giselles-ai/giselle/issues/63
https://github.com/giselles-ai/giselle/pull/77#pullrequestreview-2418059719

Changes
I have changed the default value of the following feature flags to true.
The related code for the feature flags has not been removed yet.

upload-file-to-prompt-node
web-search-node
choose-model
anthropic

Testing
Please use Vercel Feature Flags.
Other Information

I have created a new PR https://github.com/giselles-ai/giselle/pull/99, so I close this.


"npm run build" throws superjson.registerCustom is not a function

Hey, thanks for the great work on setting everything up! Wanted to give tRPC with SvelteKit a try, but the example doesn't build. It works fine when ran locally, but the build itself fails.

Thanks! 💯

Does your svelte.config.js have the following? Notably the vite part.
  kit: {
    adapter: adapter(),
    vite:
      process.env.NODE_ENV === 'production'
        ? {
            ssr: {
              noExternal: ['superjson']
            }
          }
        : undefined
  }

From this section of the docs.

Yeah. I'm using this repo. 👍

The reason I asked is because when I comment out the enitre vite property, I get the exact same issue you're getting in the screenshot. To eliminate if your original issue is caused by this, try forcing the condition to true and see if it builds:
-      process.env.NODE_ENV !== 'production'
+      true || process.env.NODE_ENV !== 'production'

That worked, thank you very much. Obviously, I would need to set NODE_ENV to production, for it to work. Duh me, thanks again!

Thanks for clearing things out, @ambiguous48!
It's in the trpc-sveltekit docs: https://github.com/icflorescu/trpc-sveltekit#configure-superjson-to-correctly-handle-decimaljs--prismadecimal
To all: do you have any suggestion on how to make this clearer?

The issue wasn't missing documentation, in my opinion, just a slight oversight not seeing that the NODE_ENV environment variable is not being set anywhere. Maybe for this example just have the build script set the environment variable?
"build": "cross-env NODE_ENV=production && svelte-kit build",

The issue wasn't missing documentation, in my opinion, just a slight oversight not seeing that the NODE_ENV environment variable is not being set anywhere. Maybe for this example just have the build script set the environment variable?
"build": "cross-env NODE_ENV=production && svelte-kit build",

I don't know, is it really necessary? I was under the impression that NODE_ENV was automatically set to production when running svelte-kit build...

The issue wasn't missing documentation, in my opinion, just a slight oversight not seeing that the NODE_ENV environment variable is not being set anywhere. Maybe for this example just have the build script set the environment variable?
"build": "cross-env NODE_ENV=production && svelte-kit build",

I don't know, is it really necessary? I was under the impression that NODE_ENV was automatically set to production when running svelte-kit build...

Yep, I thought the same. My env could be somehow broken, at least it didn't set it for me.


First integrated animation version!

Before sending your pull-request 💌 make sure :

[ ] The tests are up-to-date and your code is tested
[ ] The changes you introduce are documented in the the wiki

Merged with master, fixed some conflicts and made tests pass. Ready to merge.
Some improvements to be done in another PR.


WSL2 with /dev/dri passthrough to Docker container not working

Windows Version
Microsoft Windows [Version 10.0.22631.3880]
WSL Version
<IP_ADDRESS>
Are you using WSL 1 or WSL 2?

[X] WSL 2
[ ] WSL 1

Kernel Version
<IP_ADDRESS>-2
Distro Version
Ubuntu 22.04
Other Software
No response
Repro Steps

Verify VA-API is working within WSL2 by performing vainfo --display drm and all supported profiles for the video card should be published
Create a Ubuntu Docker container and pass through /dev/dri using sudo docker run -it --privileged --name ubuntu --device /dev/dri:/dev/dri:rwm ubuntu:22.04 bash
Inside the Ubuntu Docker container install vainfo and set the environment variables for d3d12 using the following commmands: apt update; apt install vainfo -y and export LIBVA_DRIVER_NAME=d3d12
Run vainfo --display drm within the Docker container

Expected Behavior
I expected vainfo within the container to output the same information as the WSL2 host which is the following:
libva info: VA-API version 1.14.0
libva info: User environment variable requested driver 'd3d12'
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/d3d12_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.14 (libva 2.12.0)
vainfo: Driver version: Mesa Gallium driver 23.2.1-1ubuntu3.1~22.04.2 for D3D12 (NVIDIA Quadro P1000)
vainfo: Supported profile and entrypoints
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSlice
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc

Actual Behavior
Instead what I got was a resource allocation failed error and I'm unable to use the /dev/dri device as shown below in the container output of vainfo --display drm:
libva info: VA-API version 1.14.0
libva info: User environment variable requested driver 'd3d12'
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/d3d12_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva error: /usr/lib/x86_64-linux-gnu/dri/d3d12_drv_video.so init failed
libva info: va_openDriver() returns 2
vaInitialize failed with error code 2 (resource allocation failed),exit

Diagnostic Logs
No response

WslLogs-2024-07-26_16-38-55.zip

I don't think that's enough. Look at the official guide to get VA-API inside the container:
https://github.com/microsoft/wslg/blob/main/samples/container/Containers.md#containerized-applications-access-to-vgpu-accelerated-video

@onomatopellan thank you so much, that was it!
I felt like I scoured the entire internet but wasn't able to find that, wished all this was documented on Microsoft's website instead of having to scrape through various Github issues/commits & blog posts.


Streaming to Delta Sink, Sharp Increase in Batch Time after ~36h Using Delta-1.0.0

Background
Application is using Structured Streaming to do the following every batch:

Read from Kafka Topic and decode json data using a provided Schema
repartition to reduce number of files needed later on during compaction
Write in append mode to a Delta Table in S3
In a separate (python) thread, every 15 minutes compact the data by reading in the desired amount of data from Delta Table, invoke repartition() and writing out with dataChange=False flag set

Input Rate: ~2-6k records/second depending on time of day over 25 Kafka Partitions
When Moving over to Delta-1.0.0, a "degraded state" was noticed after about 48 hours of run-time, (and since shortened to shortened to ~36 hours.). In this degraded state, batch times significantly increase, but the job continues to run. This has an almost harmonic effect, as when the batch time increases, more data has to be read, which leads to longer batch times, which leads to more backlog of data, ad infinitum.
Expected Behavior

Batch times will be relatively consistent
Behavior will be approximately the same between Delta 0.8.0 and Delta 1.0.0

Observed Behavior

Batch times spike up from ~15-20 Seconds up to ~5 minutes when using Delta 1.0.0 (and even higher on batches the require Delta Log Compaction/Checkpointing)
On job restart, batch time instantly recovers and is good for ~36 more hours
A Spark job with 10k tasks at $anonfun$gc$1 at DatabricksLogging.scala:77 shows up only when using Delta 1.0.0 that takes upwards of 4 minutes.
More complete stack trace:

org.apache.spark.sql.delta.commands.VacuumCommand$.$anonfun$gc$1(VacuumCommand.scala:239)
com.databricks.spark.util.DatabricksLogging.recordOperation(DatabricksLogging.scala:77)
com.databricks.spark.util.DatabricksLogging.recordOperation$(DatabricksLogging.scala:67)
org.apache.spark.sql.delta.commands.VacuumCommand$.recordOperation(VacuumCommand.scala:49)
org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperation(DeltaLogging.scala:106)
org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperation$(DeltaLogging.scala:91)
org.apache.spark.sql.delta.commands.VacuumCommand$.recordDeltaOperation(VacuumCommand.scala:49)
org.apache.spark.sql.delta.commands.VacuumCommand$.gc(VacuumCommand.scala:101)
io.delta.tables.execution.VacuumTableCommand.run(VacuumTableCommand.scala:69)
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:229)
org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3724)
org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:110)
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:135)
org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)

Current Debugging Done

Running Concurrent jobs in Functioning and Non Functioning Environment for comparison
Adjusting Dynamic Allocation on/off
Adjusting YARN resource calculator in case it was resource-starvation

Debugging yet-to-be-done

Run example writing to parquet instead of delta to rule out EMR/Spark versioning Issues
Move Compaction and Vacuuming to another Driver to rule out memory/GC issues due to competing jobs

Environment and Configs
Node Count/Size

Master: 1x i3.4xLarge
Core: 2-3x i3.4xLarge
Task: 0

Storage

Aws S3, no VPC Endpoint

Functioning Environment

EMR 6.1 (Spark 3.0.0)
Delta 0.8.0, scala 12
Additional Packages:

     log4j:apache-log4j-extras:1.2.17,
     org.apache.kafka:kafka-clients:2.6.0,
     org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0

Non-Functioning Environment

EMR 6.3 (Spark 3.1.0)
Delta 1.0.0, Scala 12
Additional Packages:

    log4j:apache-log4j-extras:1.2.17
    io.delta:delta-core_2.12:1.0.0
    com.qubole.spark/spark-sql-kinesis_2.12/1.2.0_spark-3.0
    org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2

Spark Configs
Application Configs

Run using:

spark-submit --deploy-mode cluster --master yarn --py-files MY_HELPERS --packages MY_PACKAGES --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog --conf spark.sql.parquet.outputTimestampType=INT96 s3://PATH_TO_MY_SCRIPT

        .set('spark.scheduler.mode', 'FAIR') \
        .set("spark.executor.cores", CORE_VALUE) \
        .set("spark.executor.memory", MEMORY_VALUE)\
        .set('spark.dynamicAllocation.enabled', 'true')\
        .set('spark.sql.files.maxPartitionBytes', '1073741824') \
        .set('spark.driver.maxResultSize', 0) \
        .set('spark.dynamicAllocation.minExecutors','3')\
        .set('spark.executor.heartbeatInterval', '25000') \
        .set('spark.databricks.delta.vacuum.parallelDelete.enabled', 'true') \
        .set('spark.databricks.delta.retentionDurationCheck.enabled', 'false') \
        .set('spark.databricks.delta.checkpoint.partSize', '1000000') \
        .set('spark.databricks.delta.snapshotPartitions', '150')

Output Configs
                .option('maxRecordsPerFile', 3000000) \
                .option('mergeSchema', 'true') \
                .option('checkpointLocation', output_location + table_name + f'/_checkpoints/{config["source_name"]}') \
                .partitionBy('HOURLY_TIMESTAMP_FIELD') \
                .start(output_location + table_name) \
                .awaitTermination()

Delta Table Configs
Non-Functioning Environment
            .property('delta.deletedFileRetentionDuration', '6 HOURS') \
            .property('delta.logRetentionDuration', '96 HOURS')

Functioning Environment

Default Settings

cc: @dennyglee following slack conversation

Thanks @mlecuyer-nd for the meticulous repro information.  As you already noted in the debugging yet-to-be done section, let's see if we can isolate whether this is related to EMR 6.3, Spark 3.1, and/or Delta Lake 1.0 that's causing what appears to be vacuum-related issues.  T

Background
I am encountering the same issue as @mlecuyer-nd where significant time is being spent in a job with the description $anonfun$gc$1 at DatabricksLogging.scala:77 during vacuum.
I created a very simple example to illustrate the problem.  I created a simple table with 2 rows of data then immediately vacuumed.  During vacuuming there was a Spark Job that was scheduled with 10K tasks and took 7 seconds.  I also noticed some other jobs that seemed to have excessive tasks given the simplicity of this example.
Description of the example is below.
Environment
I am using Spark in Local Mode via PySpark on an EC2 instance running Amazon Linux 2.  I am using Delta with a local file-system based table.  I am executing my code via a Jupyter notebook.

OS: Amazon Linux release 2 (Karoo)
CPU: EC2 Instance @ Intel Xeon Platinum 8259CL (Hypervisor: KVM), 8 cores (16 threads)
RAM: 64GB
Hard Drive Space: 400GB
Java Version: OpenJDK Runtime Environment (build 1.8.0_302-b08)
Python Version: 3.7.10
PySpark Version: 3.2.0
Spark Version: 3.2.0
Hadoop Version: 3.3.1
Delta Version: io.delta:delta-core_2.12:1.1.0

Spark Context Configuration
import os, sys
os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages=io.delta:delta-core_2.12:1.1.0 pyspark-shell'

import pyspark
from delta import *

builder = pyspark.sql.SparkSession.builder \
    .master("local[*]") \
    .appName("VacuumBug") \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
    .config('spark.databricks.delta.retentionDurationCheck.enabled', 'false') \
    .config('spark.databricks.delta.vacuum.parallelDelete.enabled', 'true')

spark = configure_spark_with_delta_pip(builder).getOrCreate()
sc = spark.sparkContext

print('python version:', sys.version)
print('pyspark version:', sc.version)
print('java spark version:', sc._jsc.version())
print(f'Hadoop version = {sc._jvm.org.apache.hadoop.util.VersionInfo.getVersion()}')

python version: 3.7.10 (default, Jun  3 2021, 00:02:01)
[GCC 7.3.1 20180712 (Red Hat 7.3.1-13)]
pyspark version: 3.2.0
java spark version: 3.2.0
Hadoop version = 3.3.1

Example
This example creates a table with only two rows and three columns, one of which is the partition column.
Raw Source Data
# test_data_a.json
{
    "name": "John",
    "age": 14
}

# test_data_b.json
{
    "name": "Mary",
    "age": 77
}

Creating the Delta Lake Table
import pyspark.sql.functions as F

TABLE_PATH = "/tmp/vacuum-bug/table"

jsonPath = "/tmp/vacuum-bug/test_data_*.json"
jsonDF = spark.read.json(jsonPath, multiLine=True)
jsonDF = jsonDF.withColumn("partitionColumn", F.lit(1))
jsonDF.printSchema()
jsonDF.show(vertical=False)

jsonDF = jsonDF.repartition(1, "partitionColumn")
print('num partitions:', jsonDF.rdd.getNumPartitions())

jsonDF \
    .write.format("delta") \
    .option("overwriteSchema", "true") \
    .mode("overwrite") \
    .partitionBy("partitionColumn") \
    .save(TABLE_PATH)

root
 |-- age: long (nullable = true)
 |-- name: string (nullable = true)
 |-- partitionColumn: integer (nullable = false)

+---+----+---------------+
|age|name|partitionColumn|
+---+----+---------------+
| 77|Mary|              1|
| 14|John|              1|
+---+----+---------------+

num partitions: 1

Filesystem After Table Creation
/tmp/vacuum-bug % ls -l
drwxr-xr-x 4 user group 4096 Dec  8 20:52 table
-rw-r--r-- 1 user group   38 Dec  8 20:23 test_data_a.json
-rw-r--r-- 1 user group   39 Dec  8 20:23 test_data_b.json

/tmp/vacuum-bug % ls -al ./table/partitionColumn=1
drwxr-xr-x 2 user group 4096 Dec  8 20:52 .
drwxr-xr-x 4 user group 4096 Dec  8 20:52 ..
-rw-r--r-- 1 user group  719 Dec  8 20:52 part-00000-ce61e66d-f0ef-4133-a07e-b89c7d10569c.c000.snappy.parquet
-rw-r--r-- 1 user group   16 Dec  8 20:52 .part-00000-ce61e66d-f0ef-4133-a07e-b89c7d10569c.c000.snappy.parquet.crc

/tmp/vacuum-bug % ls -al ./table/_delta_log
drwxr-xr-x 2 user group 4096 Dec  8 20:52 .
drwxr-xr-x 4 user group 4096 Dec  8 20:52 ..
-rw-r--r-- 1 user group 1067 Dec  8 20:52<PHONE_NUMBER>0000000000.json
-rw-r--r-- 1 user group   20 Dec  8 20:52 .00000000000000000000.json.crc

Reading Table
deltaTable = DeltaTable.forPath(spark, TABLE_PATH)
deltaTableDF = deltaTable.toDF()

print('num partitions:', deltaTableDF.rdd.getNumPartitions())
deltaTableDF.show(vertical=False)

num partitions: 1
+---+----+---------------+
|age|name|partitionColumn|
+---+----+---------------+
| 77|Mary|              1|
| 14|John|              1|
+---+----+---------------+

Vacuum Table
Lastly I do a vacuum.  This ideally would be a no-op because there are no changes/versions/etc on this table.
from delta.tables import *

deltaTable = DeltaTable.forPath(spark, TABLE_PATH)
deltaTable.vacuum(0)

/tmp/vacuum-bug % ls -al table/partitionColumn=1
drwxr-xr-x 2 user group 4096 Dec  8 20:58 .
drwxr-xr-x 4 user group 4096 Dec  8 20:58 ..
-rw-r--r-- 1 user group  719 Dec  8 20:58 part-00000-7c2ad9fb-8191-42a9-9d55-19f8c5acc040.c000.snappy.parquet
-rw-r--r-- 1 user group   16 Dec  8 20:58 .part-00000-7c2ad9fb-8191-42a9-9d55-19f8c5acc040.c000.snappy.parquet.crc

/tmp/vacuum-bug % ls -al table/_delta_log
drwxr-xr-x 2 user group 4096 Dec  8 20:58 .
drwxr-xr-x 4 user group 4096 Dec  8 20:58 ..
-rw-r--r-- 1 user group 1067 Dec  8 20:58<PHONE_NUMBER>0000000000.json
-rw-r--r-- 1 user group   20 Dec  8 20:58 .00000000000000000000.json.crc

Spark Jobs
Below is a screenshot from the Spark Web UI showing the jobs that were completed for each of the steps in this example.

Jobs 0-5 were executed for the creation of the table
Jobs 6-12 were executed for reading the table
Jobs 13-18 were executed when the table was vacuumed

Problem/Issue
If you look at the jobs, it appears that for both reading and vacuuming, there are numerous tasks being spawned that are unnecessary.

During reading, Jobs 8 and 10 spawned 50 tasks.  This seems very strange due to the fact that this table is so small and has only a single version.
During vacuuming, Job 13 created 10K tasks, Jobs 15 and 18 created 200 tasks and Job 16 created 50 tasks.  This seems overly excessive.  Job 13 took 7 seconds to complete!

I would also like to point out that another issue from 2020 that was previously closed showed similar behavior.
https://github.com/delta-io/delta/issues/387#issuecomment-614610749

Thanks very much for providing the additional details @inh3 - we will continue the investigation as well!

Note, I did a quick test following @inh3 excellent instructions and we may have separate issues - one that @inh3 calls out as noted in #387 and the issue that you're still facing @mlecuyer-nd.
Test Reproduction
Follow the exact same steps as @inh3 noted using Delta Lake 0.8.0 (Spark 3.0.0) and Delta Lake 1.1 (Spark 3.2) which matches @mlecuyer-nd configuration. Note, I'm running this on my local box vs. EC2 instance but this is moot for this discussion.
Using Delta 1.1
The results are similar to yours @inh3 per the following view with 21 jobs generated following the same steps and a 10s job with 10,216 tasks.

Using Delta 0.8.0
What is interesting here is that because of the recursive gc as noted in #387, there are 210 jobs (10x) spawned.

But when diving deeper, it's still the same 10s and 10,000 tasks that is happening.

As was noted in #387, vacuum can be expensive  because it does the following

List all the files in the directory (recursively if partitioned)
Find out all files needed by all the retained table versions.
Join these to find which files are not needed and then delete them.

While I can understand the consternation of this vacuum taking longer than desired, it appears its the same issue in terms of many tasks / duration for vacuum ala #387.  Perhaps we can get a PR or create a separate issue for more efficient vacuum :)
Saying this, @mlecuyer-nd issue may be different in that the same code results in an extended delay for 1.0 vs. 0.8.  Note, this could be an artifact of the expense of vacuum but it should be the same (or better) in 1.0 vs. 0.8.  We still have more digging to do, eh?!

same  Issues！！！

Thank you for the detailed explanation! the 10000+ tasks job is performing the recursive listing of the files/directories under the delta table. The number of tasks is controlled by spark.sql.sources.parallelPartitionDiscovery.parallelism spark conf which is by default 10000. This should explain the 10000 tasks stage. @dennyglee IMHO, we should default this to a lower value than 10000.
@yangyingjie this was a default behavior even before delta 1.0.0 and this should not impact your streaming query performance. Could you please provide more input about the vacuum and compaction jobs that you run concurrently with the job appending to your delta table?

@JassAbidi Awesome call out - you're right, we probably should provide better guidance on using this.  For example, in my preceding example of when using Delta 1.1, using:
spark.conf.set("spark.sql.sources.parallelPartitionDiscovery.parallelism", "10")
deltaTable.vacuum(0)

The vacuum task completed in 0.4s with 226 tasks (ssspp: 10) vs. 8s with 10,216 tasks (ssspp: 10000).
We would most likely call out this change should be done just prior to the vacuum command and then reversing this (i.e. go back to the default of spark.conf.set("spark.sql.sources.parallelPartitionDiscovery.parallelism", "10000")) so it would not impact other Spark jobs.

@JassAbidi That's very interesting, and thanks for the call out! While you didn't tag me, I'll provide the details of my compaction/vacuum as well, as I left that out of the original.
Here's the compaction:
spark.read.format('delta') \
                    .load(table_location) \
                    .where(part_statement) \
                    .repartition(4) \
                    .write \
                    .format('delta') \
                    .mode('overwrite') \
                    .option('replaceWhere', part_statement) \
                    .option('dataChange', 'false') \
                    .option('maxRecordsPerFile', '3000000') \
                    .save(table_location)

and the Vacuum is:
spark.sql(f"VACUUM delta.{table_location} RETAIN {24*7}  hours")

thanks @mlecuyer-nd - QQ if you were to reduce the parallelism for just the vacuum job, I'm wondering if there is a different result for Delta 0.8 and Delta 1.0?

Including #886 as a potential solution to this

@dennyglee Sorry I haven't had time to run those test cases for ya. I attempted to remove the vacuuming for a test case before the holidays but the upstream source wasn't as stable (was going based off of s3 replication for the delta table for a dev environment).
Will hopefully have time #886 to test that change once it goes live

Same here... any news on how to solve/mitigate this?
Thanks!

Hi @pedrosalgadowork - as we continue working on this - please check out https://github.com/delta-io/delta/issues/859#issuecomment-995454556 on a potential mitigation.  Thanks!

Hi @pedrosalgadowork - as we continue working on this - please check out #859 (comment) on a potential mitigation. Thanks!

Thanks @dennyglee!


Add ES module option.

Adds an --esm switch with documentation in both the README and the app's usage message.  When this switch is active, the bin/www (see below about the name of this file), app.js, and routes files all use ES module import/export syntax instead of Common JS require() syntax.  Other small code modifications in support of this change are made in --esm mode, such as adding .js to some filename strings, defining a __dirname variable, and adding 'node:' to Node library references (I believe that this requires the end user to be running Node version 16 or greater, which hopefully isn't a problem; it's easy to back out the change if it is a problem). One change that affects the files generated in non-esm mode is that the generated package.json file now always includes a 'type' field that has value of either 'module' when the --esm switch is active or 'commonjs' by default.
The generator code structure has been modified by adding a template/mjs folder in parallel with the existing template/js folder. The mjs folder contains the ESM versions of the example route files.  This means that any future changes to the example route files should be made in two places to maintain compatibility between the CJS and ESM. versions. An alternative to the parallel folder structure would be to generate the route files from ejs templates rather than to follow the existing approach of copying them verbatim.  I went with the simpler existing approach under the assumption that the example route files will change very infrequently if at all.
Testing of --esm has been added to test/cmd.js.  Some of this testing failed when attempting to load bin/www due to the lack of file extension. So, when --esm is active this file is named bin/www.js.  The --esm tests are based on the (no-args) tests but do not repeat the directory-oriented tests:

I'm not sure why 13.x fails when 12.x succeeds, but I'm guessing that it has something to do with 13.x not having LTS. Anyway, I'm pretty sure that the 15.x failure is not related to anything in this PR, because I see the same Python "missing parentheses" error when I run tests on the master branch.  For what it's worth, I think that the problem is that some of the Python code being run as part of the test is written in Python 2 syntax that is incompatible with Python 3 while the tests (at GitHub and on my machine) are being run using a Python 3 interpreter.

The documentation has been updated to state that Node 14.x or higher is required for the --esm switch, and the --esm switch should now lead to an error exit for all Node versions prior to 14, The only tests run for these earlier versions are to check for an error exit and for appropriate error messages. I believe that the esm automated tests will now pass for all Node versions, with versions 14 and higher running the 12 tests shown in the earlier image.  However, testing with Node 15.x and higher still produces errors on my machine related to sass, but this is true on the master branch as well.
The files generated in --esm mode now use updated JavaScript syntax, with const replacing var and arrow functions replacing function expressions in callbacks.

Am I correct in thinking that all of the PR test failures also occur when testing the master branch, or do I need to do some more work on the PR related to automated tests?

Am I correct in thinking that all of the PR test failures also occur when testing the master branch, or do I need to do some more work on the PR related to automated tests?

I think that there are some PRs with solutions for the CI, let me check. Probably we should merge them first.

type module and the node: prefix are both not required, and type module specifically should be avoided. ESM should be .mjs.

type module and the node: prefix are both not required, and type module specifically should be avoided. ESM should be .mjs.

Agreed, neither is required.  The point of the PR is to provide an option to those who would prefer to begin a new project as ESM and to use other newer Node/JavaScript features such as the node: prefix and ES6 syntax.

The node: prefix works in CJS also, and is not required in either; I'm not sure why it would be useful to conflate that with "use ESM"; similarly, type module is harmful since it violates what .js means, so express should be actively pushing people to use .mjs for ESM, since it's the standard extension for that format.

@drjeffjackson do you have any plans to work on making the tests in the pipeline to pass?

@ljharb Thanks for the additional detail; that's helpful. Regarding your observation:

The node: prefix works in CJS also, and is not required in either; I'm not sure why it would be useful to conflate that with "use ESM"

You're absolutely right. My original goal was to generate an ESM version of the skeleton code, but in implementing this I realized that this felt like a halfway measure and that what would be even better would be to generate an ES6 version of the code. Despite my development going in that direction, I still referred to this as an ESM mod.  That was an oversight, and I appreciate your pointing this out.  I'll be pushing a commit renaming the switch to --es6.
However, regarding your claim that:

similarly, type module is harmful since it violates what .js means, so express should be actively pushing people to use .mjs for ESM, since it's the standard extension for that format.

The Node v21 Modules: Packages API says:

This flag [ --experimental-default-type ] currently defaults to "commonjs", but it may change in the future to default to "module". For this reason it is best to be explicit wherever possible; in particular, package authors should always include the "type" field in their package.json files, even in packages where all sources are CommonJS. Being explicit about the type of the package will future-proof the package in case the default type of Node.js ever changes, and it will also make things easier for build tools and loaders to determine how the files in the package should be interpreted.

There is no hint here or anywhere else in the Node docs, as far as I can tell, that type: 'module' is something to avoid.  To the contrary, if there is a hint of anything it is that there is a real possibility that the default type of Node will change to module at some point in the future! There is also a relevant implicit assumption here that many of the files in a package, whether a package of type module or type commonjs, will have the .js extension: "Being explicit about the type of the package will ... make things easier ... to determine how the files in the package should be interpreted." If all of the files in a module type package had the .mjs extension, the extension by itself would determine how to interpret the files and the type field would be superfluous. So, the assumption of the quote is that the files matching the package type will likely have .js extensions and that it's therefore good practice to specify the package type, whatever it is.
In addition, a little farther down in the Modules: Packages page we find this:

The .mjs and .cjs extensions can be used to mix types within the same package:

Within a "type": "module" package, Node.js can be instructed to interpret a particular file as CommonJS by naming it with a .cjs extension (since both .js and .mjs files are treated as ES modules within a "module" package).

Within a "type": "commonjs" package, Node.js can be instructed to interpret a particular file as an ES module by naming it with an .mjs extension (since both .js and .cjs files are treated as CommonJS within a "commonjs" package).

Again, there's no hint here that anything is violated by using a "type": "module" package containing .js files that contain ESM code, any more than there is anything wrong with using a "type": "commonjs" package containing .js files that contain CJS code.
In short, I've looked, but I haven't found any support for your claim while I have found what seem to me to be strong indicators to the contrary. However, maybe I've missed something. I'd be interested in any pointers you have to reputable recent recommendations that "type": "module" be avoided in new projects.

@drjeffjackson do you have any plans to work on making the tests in the pipeline to pass?

@joeyguerra: I believe that my PR code passes all of the tests that I added for it and causes no problems with existing tests. That is, I believe that all of the test failures are due to other code that, as far as I can tell, also fails on the master branch.  @UlisesGascon is I believe looking into fixing these tests, so no, I have no plans currently to correct these problems since they're not problems with my code/tests.
That said, if it is a problem for others to fix the failing tests and if all tests need to pass before my PR can be approved, it would not be hard for me to submit a separate PR that skips the failing tests on a version-specific basis, such as adding code that for v0.10 skips the pug tests that fail.

@drjeffjackson I pulled down the PR to investigate and found a bug in the app.js.ejs template. createError doesn't exist in the --esm -> "should generate a 404" test case.

@drjeffjackson I pulled down the repo for this PR and investigated the issues. I believe the hypothesis about python2 is correct. My initial thought is to install python 2 with pyenv on the ci/cd runner before the npm install step.
I also found a bug in the app.js.ejs file. The 404 test case under the esm test suite fails because "createError is not defined". Here's the diff:

@drjeffjackson I pulled down the PR to investigate and found a bug in the app.js.ejs template. createError doesn't exist in the --esm -> "should generate a 404" test case. createError is only defined when --esm is not used. Adding the following causes the test to pass:
<% if (esm) { -%>
import express from 'express';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
<% Object.keys(modules).sort().forEach(function (variable) { -%>
import <%- variable %> from '<%- modules[variable] %>';
<% }); -%>
// The following line fixes the test.
import createError from 'http-errors';

I am not sure what you are looking at, @joeyguerra? Here is what I see on my branch at GitHub:

@drjeffjackson you're correct. I had removed lines 2 through 4 when troubleshooting the "http-errors not found" issue in the node 15 pipeline run.

@drjeffjackson you're correct. I had removed lines 2 through 4 when troubleshooting the "http-errors not found" issue in the node 15 pipeline run.

Ah, okay. So the issue is in the master-branch code, which imports createError conditioned on view. My code does the same thing,  and I think that you're saying that createError needs to be imported unconditionally in both versions.  Anyway, sure, I can make that change in my code. Would you want me to change the original code to match as well?

@drjeffjackson I think the app.js.ejs code should "work" as it stands in this PR. I had deleted those lines as I was troubleshooting the missing http-errors module error, but forgot to put it back once I resolved it by installing Python 2.

@drjeffjackson the docs are written by folks with an agenda. I was on the modules working group, and i maintain module tooling, and I’m on the language committee, and i can assure you that .mjs is the standard extension for ESM; changing the default of the “type” field won’t affect anyone who uses .mjs - only those who use .js. It’s not that using type module “violates” anything, it’s that it’s sole purpose is to subvert why “.js” means - from CJS to ESM - and that can be bypassed simply by using the standard ESM file extension.

2 different viewpoints to the mjs debate.
mdn

For learning and portability purposes, we decided to keep to .js.

V8 team

for Clartity for the reader and the runtime/tooling, use .mjs.

Btw, PR #314 adds Python 2 to the pipeline runner which I think would make all the pipeline tests in this PR pass.

@drjeffjackson the docs are written by folks with an agenda. I was on the modules working group, and i maintain module tooling, and I’m on the language committee, and i can assure you that .mjs is the standard extension for ESM; changing the default of the “type” field won’t affect anyone who uses .mjs - only those who use .js. It’s not that using type module “violates” anything, it’s that it’s sole purpose is to subvert why “.js” means - from CJS to ESM - and that can be bypassed simply by using the standard ESM file extension.

@ljharb Impressive resume, but I notice that you don't seem to have anything to point to to back up your claim about the type field other than your own informed opinion.  I like to think that I have an impressive resume and informed opinions as well, and in my opinion new code should be written using ES6+, including ES modules. It's simply a better way to code. Also in my opinion, setting type to module is simply a way of encouraging that desirable behavior in new projects.
As for subverting the meaning of .js, .js means that a file contains JavaScript, pure and simple.  Yes, if you look at old Node code, you are going to expect to see the CJS module system used in many .js files.  But the question of interest is, what module system should we expect to see used in new Node .js files?  I don't see any problem with saying that we don't know what to expect but that in a well-written package we will be able to look at package.json to find out.  How is this subversive?

JavaScript has two parse goals, Script and Module, and since web browsers don't understand file extensions at all, the only runtime uses of .js before ESM was usable in any engine was the Script goal in browsers, and CJS in node. Additionally, CJS isn't just in "old" node code, it's the vast majority of npm's downloads.
The extension should be telling you what module system it is, and if you want to clearly indicate ESM, you'd use .mjs. You shouldn't have to look at package.json to know, and moving the file elsewhere shouldn't lose this out-of-band information when the proper in-band mechanism exists - the extension.

JavaScript has two parse goals, Script and Module, and since web browsers don't understand file extensions at all, the only runtime uses of .js before ESM was usable in any engine was the Script goal in browsers, and CJS in node. Additionally, CJS isn't just in "old" node code, it's the vast majority of npm's downloads.
The extension should be telling you what module system it is, and if you want to clearly indicate ESM, you'd use .mjs. You shouldn't have to look at package.json to know, and moving the file elsewhere shouldn't lose this out-of-band information when the proper in-band mechanism exists - the extension.

I agree that using .mjs has certain advantages. It also has some disadvantages, as noted in the MDN article.  If the module type can stay in, I can generate files with .mjs extensions.

btw i think "es6" isn't a good name; ES6 is ES2015, which was 9 years ago, and there's lots of module features added since then - and node didn't ship ESM support until 2020. if it's a broad "use one person's idea of modern practices" then a generic name like "modern" is probably more appropriate; if it's named something like "ESM" it should omit subjective opinions.

I tested this with the following steps (pulled @drjeffjackson repo and esm branch)
npm install
npm link
express myapp --es6
cd myapp
npm install
DEBUG=myapp:* npm start

Visited http://localhost:3000 in my browser.

Page was loaded as expected
Looked in myapp and saw ES6 style javascript files and code

everything LGTM. Let me know if there's anything else I should look for.

I tested this with the following steps (pulled @drjeffjackson repo and esm branch) [...]

Thanks, @joeyguerra! One other test would be visiting http://localhost:3000/users, which should respond with the text "respond with a resource". And you should also see ES6 style in bin/www, routes/index, and routes/users.

Looks really promising. When is it going to merge onto the actual package?


 U: .

 U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3053/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 10s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+0 :ok:
shelldocs
0m  0s
Shelldocs was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

_ Patch Compile Tests _

-0 :warning:
shellcheck
0m  1s
The patch generated 10 new + 23 unchanged - 15 fixed = 33 total (was 38)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

_ Other Tests _

+1 :green_heart:
asflicense
0m 16s
The patch does not generate ASF License warnings.

2m 57s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3053/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3053

Optional Tests
dupname asflicense shellcheck shelldocs

uname
Linux 1ae3ee9f020e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 59ec375d67

shellcheck
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3053/1/artifact/yetus-general-check/output/diff-patch-shellcheck.txt

Max. process+thread count
51 (vs. ulimit of 30000)

modules
C: . U: .

 U: .

 U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3053/2/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 32s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+0 :ok:
shelldocs
0m  0s
Shelldocs was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

_ Patch Compile Tests _

-0 :warning:
shellcheck
0m  1s
The patch generated 9 new + 23 unchanged - 15 fixed = 32 total (was 38)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

_ Other Tests _

+1 :green_heart:
asflicense
0m 18s
The patch does not generate ASF License warnings.

3m 23s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3053/2/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3053

Optional Tests
dupname asflicense shellcheck shelldocs

uname
Linux f87e651b1e9e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 59ec375d67

shellcheck
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3053/2/artifact/yetus-general-check/output/diff-patch-shellcheck.txt

Max. process+thread count
48 (vs. ulimit of 30000)

modules
C: . U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3053/2/console

versions
git=2.17.1 maven=3.6.3 shellcheck=0.4.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Mmmh I might have misunderstood the "localhost" mode.
My intention is to be able to graceful_restart a local RegionServer in a secure environment where we can't use ssh to restart remotely.
However, seems the localhost mode is actually meant to be used by pseudo-distributed mode:

https://issues.apache.org/jira/browse/HBASE-15924?focusedCommentId=16410243&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16410243
https://issues.apache.org/jira/browse/HBASE-20130

And in localhost mode, regionservers.sh uses hard-coded ports:
https://github.com/apache/hbase/blob/d74ae15fa7a110d3e2b35bf53d8823ba0b1bb33b/bin/regionservers.sh#L61-L64
This is very confusing, because if someone has configured other ports and restart in localhost, it will see the ports changing.
Let me rethink about a proper solution or just leave the original proposed change https://github.com/apache/hbase/pull/2973/files (pushed by @saintstack in this commit https://github.com/apache/hbase/commit/59ec375d679dcbf1701a92d4206ecc676a8fb069).
So please don't merge this yet (I may close it an open with a new solution).

Sorry for so many PRs, proceeding here: https://github.com/apache/hbase/pull/3054


Update explore-intro.html

The kubernetes-bootcamp pod exposes its webserver not on port 80 but on port 8080.

As this issue is already resolved in #41929, we can close this PR
/close


grootfs cleanup does not work correctly

Describe the bug
We observed that the contents of /var/vcap/data/store were not cleaned up when a cell pod restarts. This led to garden errors like guardian.create.volume-creator.image-plugin-create.grootfs.create.groot-creating.making-image.overlayxfs-creating-image.applying-quotas.run-tardis.tardis.set-quota.set-quota.setting-quota-to-project-id-failed , caused by stale information about projectids being carried over from a previous 'life' of the data directory from a previous pod.
Looking at the bpm-pre-start-garden container logs we can see an error where the clean did not work correctly in the pre-start
To Reproduce
Restart a cell pod and then look at the contents under /var/vcap/data/grootfs of the replacement pod that starts. You will see contents that are older than the lifetime of the new pod, demonstrating that the cleanup did not succeed. You can also look at the bpm-pre-start-garden logs and you will see an error from the find command, saying "paths must precede expression". For example:
kubectl -n kubecf logs diego-cell-3 bpm-pre-start-garden
+ /var/vcap/jobs/garden/bin/bpm-pre-start
find: paths must precede expression: bin
Usage: find [-H] [-L] [-P] [-Olevel] [-D help|tree|search|stat|rates|opt|exec] [path...] [expression]
2020-08-17T16:58:16.630871991Z - running thresholder
2020-08-17T16:58:16.637805608Z - done
real	0m0.059s
user	0m0.026s
sys	0m0.010s

Expected behavior
After a pod restarts, there should be no error in the bpm-pre-start-garden log and the /var/vcap/data/grootfs contents should all be created since the pod restarted.
Environment

CF Version: 13.9
Component/Plugin Name
Others

Additional context
Here is the code in question: https://github.com/cloudfoundry-incubator/kubecf/blob/master/bosh/releases/pre_render_scripts/diego-cell/garden/jobs/patch_pre-start.sh#L18
and here is the change that added this grootfs cleanup: https://github.com/cloudfoundry-incubator/kubecf/commit/3d0562919480c6b40bf09a21221f625f72d46e0c
slack discussion: https://cloudfoundry.slack.com/archives/CQ2U3L6DC/p1597411497472300

should be fixed by #1264


Added Animated Login Form

Fixes Issue🛠
Closes #93
Description👨💻
Added an animated login form
Type of change📄

[X] New feature (non-breaking change which adds functionality)

Checklist✅

[X] My code follows the style guidelines of this project
[X] I have performed a self-review of my own code
[X] I have commented my code, particularly in hard-to-understand areas
[X] I have added demonstration in the form of GIF/video file
[X] I am an Open Source Contributor

Screenshots/GIF📷

Attractive, Merged!! 🎉🥳 @rishicds


Nvidia: Add tips to fixing suspend issues

First PR ever, so here we go:
Nvidia's proprietary drivers include some systemd services to aid with suspend/hibernate and they aren't enabled by default in any of the distros I've seen, and their userspace driver only restores essential memory on wakeup, leading to graphical corruption and crashes, unless their driver is loaded with a specific parameter.

Should be the same on debian and fedora based distros as well except for driver package name.
From my brief experience on Nixos a couple months ago, the equivalents for these are:
boot.kernelParams = [ "nvidia.NVreg_PreserveVideoMemoryAllocations=1" ];
...
#false by default
hardware.nvidia.powerManagement.enabled = true

#false by default, but good to add in case of upstream changes
hardware.nvidia.open = false

Should I include these in the nvidia page as well?

Sure, why not.

Done


Why is CVSS limited to CVSS v3.x?

CVSS_V3 | A CVSS vector string representing the unique characteristics and severity of the vulnerability using a version of the Common Vulnerability Scoring System notation that is >= 3.0 and < 4.0 (e.g."CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:C/C:H/I:N/A:N").
Can we please either:

add CVSS_V2 tag for CVSS v2 data (e.g. historical stuff)
change CVSS_V3 to CVSS and simply specify the CVSS version in the string (as generally done). This also solves the "what about CVSS v4 problem in future?

Please propose a PR for the CVSS_V2 tag! That should be a very straightforward tag to add.

Is there documentation on this? E.g. what do I need to actually change?
https://github.com/ossf/osv-schema/blob/main/validation/schema.json
https://github.com/ossf/osv-schema/blob/main/docs/schema.md
anything else?

That should be it.

Ok done https://github.com/ossf/osv-schema/pull/86

This one: https://github.com/ossf/osv-schema/pull/87

added in https://github.com/ossf/osv-schema/pull/87


Fix visual filename error when behave feature contains :

Coverage remained the same at 91.434% when pulling db2771ac8dee8dbe26e49895ce91bbfdc31e7958 on fix/visual_name into 1d5ad70bd495fc8ffa1afffddcf6dbe6a5036a24 on master.


[QUESTION] How to get follower / following user ids instead of screen names

Command Ran
    config = twint.Config()
    config.Username = SCREEN_NAME
    config.Limit = 20
    config.Hide_output = True
    config.Store_object = True
    #config.User_full = True # seems to be a lot slower
    config.Store_object_follow_list = []
    print(config)

    twint.run.Following(config)
    # takes a few requests
    # seeing the occasional "CRITICAL:root:twint.feed:Follow:IndexError"

    print("FOLLOWS LIST:", twint.output.follows_list) #> []
    print("USERS LIST:", twint.output.users_list) #> []
    print("STORED OBJECT FOLLOWS LIST:", config.Store_object_follow_list) #> yay, a list of screen names!

Description of Issue
The code above provides a list of screen names who the user follows in the config.Store_object_follow_list, but is there a way to return their user ids instead?
I tried a few approaches setting config.User_full = True, but it seems to be a lot slower. And I was hoping to use the faster approach.
Environment Details
Mac OS, inside anaconda environment with Python 3.7

I also tried a few approaches using config.Custom like config.Custom = ['id'] and config.Custom["user"] = ['id'] but neither seems to work.

Retrieving only the user id's is not currently possible without retrieving the full profile of the users being followed by or are following the user specified in your search.
The technical reason behind this is that Twitter returns permalinks for these lists.  Specifying User_full = True will instruct twint to retrieve those permalinks and populate the output.users_list list with the profiles.


Empty filter crashes query

Hey all, really love this extensions, as it makes my dev process a breeze 🍃
When a collection is queried in GraphQL and an empty filter is passed, a syntax error appears.
I can only reproduce this with the @supabase/cli.
To Reproduce
Steps to reproduce the behavior:

Setup a local supabase
Create any table
Query like this:

query GetProjects {
  projectCollection(filter: {}) {
    edges {
      node {
        id
        name
      }
    }
  }
}

Receive the following error / response:
{
  "data": null,
  "errors": [
    {
      "message": "syntax error at or near \")\""
    }
  ]
}

Expected behavior
Should just behave like no filter was passed.
Example:
query GetProjects {
  projectCollection {
    edges {
      node {
        id
        name
      }
    }
  }
}

Response:
{
  "data": {
    "projectCollection": {
      "edges": []
    }
  }
}

Versions:

@supabase/cli 1.1.6
PostgreSQL: [e.g. 14.3]

Additional context
The remote version, which is working, is running on<EMAIL_ADDRESS>– this might be the difference.

I wasn't able to reproduce this on the platform or on the latest CLI version 1.4.7
could you please confirm that you're still experiencing the issue?
This was the test setup
create table account(
	id int primary key,
	name text
);

insert into account(id, name) values (1, 'a'), (2, 'b'), (3, 'c');

{
  accountCollection(filter: {})
  {
    edges {
      node {
        id
      }
    }
  }
}

result:
{
  "data": {
    "accountCollection": {
      "edges": [
        {
          "node": {
            "id": 1
          }
        },
        {
          "node": {
            "id": 2
          }
        },
        {
          "node": {
            "id": 3
          }
        }
      ]
    }
  }
}

@olirice Seems to be resolved with the latest CLI version now! Thanks :)

Also in our real application case 👏


Improve documentation

Codecov Report

Merging #18 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #18   +/-   ##
=========================================
  Coverage     92.15%   92.15%
  Complexity       53       53
=========================================
  Files             6        6
  Lines           153      153
  Branches         17       17
=========================================
  Hits            141      141
  Misses            7        7
  Partials          5        5

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 217be64...b68274c. Read the comment docs.


Redacted message content in notifications email

I can't find any way to remove the message content from notification emails server-wide. Would that be possible to add similar to how you can do it with the mobile app notifications?
I know it is possible for each user to set it in the settings in the zulip client, but it would be great if it was a way to enforce it from the server side.

@Alge thanks for the feedback.  I think it could make sense to add a setting similar to PUSH_NOTIFICATION_REDACT_CONTENT that basically overrides the user-level setting; is that what you have in mind?
(I assume the motivation is a security policy thing?)

I want to add a vote for this feature. We have few enough users at the moment that I can trust them all to uncheck the "Include message content in message notification emails" box, but I'd like to not have to worry about it. A setting similar to PUSH_NOTIFICATION_REDACT_CONTENT would work well for us.
I can't speak for others, but our motivation on this is security policy; we share some things on Zulip that would not be appropriate to send over email (e.g., passwords)


Bump versions

Bump identity.inbound.provisioning.scim2.version 1.6.16 -> 1.6.17
identity.org.mgt.version 1.0.23 ->1.0.28

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/2597950565

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/2597950565
Status: failure


Flutter 2 update

Hi, any chance of updating this package to support Flutter 2? Looks like it initially needs to update intl from 0.16.0 to 0.17.0.

Just as importantly, null-safety needs to be implemented.

I guess this library is unmaintained?

Sorry I was very busy. Just upgrade to null safety.
Feel free to PR and contribute

@iot-d Perhaps you could merge the changes from your fork here?

I have upgraded to null safety yesterday.

@ryanhz Great! Thanks for pointing that out :)
Could you update the examples in the readme? I'm getting the A value of type 'String?' can't be returned from the function 'about' because it has a return type of 'String'. error when using _bundle.format — is the preferred way of dealing with that just go append ! at the end of every format call? (It doesn't seem like great practice to me, but that seems to be the way it's architectured, right?)

(Nevermind, forgot about the separate l10n_provider file — I'll be changing all instances of String into String?, then.)

Oh, I fogot readme.   This is the result of nullable.  Actually it return the same thing, it can return null previously as well (for example you pass in the wrong message id).
I suggest add a wrapper function to deal with null, for example, return ! id ! as default return value to indicate a missing id.


director_uuid is not updated, when updating Bosh version

We set the directory_uuid in the deployment yml via a environment variable. The environment variable has been set by us correctly.
director_uuid = ENV["JB_DIRECTOR_UUID"]
director_uuid: <%= director_uuid %>

The problem is that the directory_uuid is not updated in the generated .deployments file.
The solution is to set the new directory_uuid manually in the generated .deployment file or just delete the file.
Bosh Version 260.6

Closed here. Reopened at right place: https://github.com/cloudfoundry-incubator/bosh-workspace/issues/126


[BUG] engine integration test: test_snapshot_tree_basic fails on ARM64

Describe the bug
test_snapshot_tree_basic occasionally fails on ARM64.  The line of the test that fails is https://github.com/longhorn/longhorn-engine/blob/870f06d50ece703aed65eb4db617c013bbdf3cd0/integration/common/core.py#L303.  This seems to indicate that the purge progress is over 100%.  I think the problem is that we are running commands to remove snapshots then purge snapshots.  The commands construct gRPC calls to remove snapshots and the last remove snapshot command might not finish before the snapshot purge.  We could add a sanity check to make sure this line (https://github.com/longhorn/longhorn-engine/blob/870f06d50ece703aed65eb4db617c013bbdf3cd0/pkg/sync/rpc/server.go#L1114 doesn't set the progress to over 100 or make the function that sets the progress be able to handle rm snapshots while it is executing.
To Reproduce
Steps to reproduce the behavior:

Perform build on ARM64
See that test_snapshot_tree_basic fails.

Expected behavior
test_snapshot_tree_basic should pass.

Closing because the test failed not shown up anymore.


[WIP][FLINK-27273] Zookeeper HA support

⚠ This PR is WIP, my goal is to get feedback as early as possible

What is the purpose of the change
https://issues.apache.org/jira/browse/FLINK-27273
This PR makes sure that Flink applications which use Zookeeper-based HA can be deployed via the Kubernetes Operator.
It wasn't previously possible mainly due to the lack of proper clean-up of Zookeeper data which could cause unexpected behaviour in Flink applications, as mentioned in the ticket.
Brief change log

JobGraph-related data is now properly cleaned in Zookeeper (same behaviour as when using Kubernetes HA)
Code has been updated to support both Kubernetes and Zookeper-based HA settings. All features which worked with Kubernetes HA (such as JM deployment recovery, unhealthy job restarts or rollbacks) should also work with Zookeeper HA as well.

Verifying this change

TODO: add unit tests!
Tried some scenarios in our Kubernetes environment:

upgradeMode: savepoint:

[x] Successfully deployed a Flink application with upgradeMode: savepoint and Zookeeper HA enabled
[x] Updated the FlinkDeployment object, verified that the JobGraph data was successfully deleted in Zookeeper and that the new version of the application was successfully rolled out
[x] Manually deleted the JobManager Kubernetes Deployment, verified that the Operator was able to recover it (with kubernetes.operator.jm-deployment-recovery.enabled and kubernetes.operator.job.upgrade.last-state-fallback.enabled set to true) and that the application restarted successfully (this process relies on HA metadata too)

upgradeMode: last-state:

[x] Successfully deployed a Flink application with upgradeMode: last-state and Zookeeper HA enabled
[x] Updated the FlinkDeployment object, verified that the JobGraph data was successfully deleted in Zookeeper and that the new version of the application was successfully rolled out
[x] Manually deleted the JobManager Kubernetes Deployment, verified that the Operator was able to recover it with kubernetes.operator.jm-deployment-recovery.enabled set to true and that the application restarted successfully (this process relies on HA metadata too)

[x] Set kubernetes.operator.cluster.health-check.enabled to true and deployed a Flink application which was continuously failing and restarting. Once the restart threshold (kubernetes.operator.cluster.health-check.restarts.threshold) was hit, the Operator was able to successfully validate that the Zookeeper HA metadata exists and restarted the job
[x] Set kubernetes.operator.deployment.rollback.enabled to true and deployed a Flink application which was in CrashLoopBackOff. Once the kubernetes.operator.deployment.readiness.timeout passed, the Operator was able to successfully validate that the Zookeeper HA metadata exists and rolled back the job to the previous version.
TODO: anything else?

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changes to the CustomResourceDescriptors: no
Core observer or reconciler logic that is regularly executed: yes

Documentation

Does this pull request introduce a new feature?: yes
If yes, how is the feature documented?: Updated the documentation where applicable

I think from an initial review this looks pretty good. We just have to ensure that we handle the Zookeeper dependency as optional as this is the less frequent integration :)

Thanks a lot @gyfora!
Regarding:

We just have to ensure that we handle the Zookeeper dependency as optional

The dependency was actually already there! I didn't have to add anything.
I looked into it and all Curator classes are coming from the flink-clients package which is listed as a dependency in pom.xml here.

$ mvn dependency:tree -pl 'flink-kubernetes-operator'
[...]
[INFO] +- org.apache.flink:flink-clients:jar:1.15.3:compile
[INFO] |  +- org.apache.flink:flink-runtime:jar:1.15.3:compile
[INFO] |  |  +- org.apache.flink:flink-rpc-core:jar:1.15.3:compile
[INFO] |  |  +- [...]
[INFO] |  |  +- org.apache.flink:flink-shaded-zookeeper-3:jar:3.5.9-15.0:compile

Thanks a lot @gyfora!
Regarding:

We just have to ensure that we handle the Zookeeper dependency as optional

The dependency was actually already there! I didn't have to add anything.
I looked into it and all Curator classes are a transitive dependency of the flink-clients package which is listed as a dependency in pom.xml here.
$ mvn dependency:tree -pl 'flink-kubernetes-operator'
[...]
[INFO] +- org.apache.flink:flink-clients:jar:1.15.3:compile
[INFO] |  +- org.apache.flink:flink-runtime:jar:1.15.3:compile
[INFO] |  |  +- org.apache.flink:flink-rpc-core:jar:1.15.3:compile
[INFO] |  |  +- [...]
[INFO] |  |  +- org.apache.flink:flink-shaded-zookeeper-3:jar:3.5.9-15.0:compile

Hm, thats convenient :)

I made some progress on the PR but still need to implement this HA data deletion logic for Zookeeper:
https://github.com/apache/flink-kubernetes-operator/blob/d0b769c0d80fee526f5abda42673ae4c856fee3c/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L931-L938


[Feature Request] Add ability to quickly add a new site

Add a quick add feature where you click a button to enter in class names and a regex to match the site url.

Really like this one.
This would be the start of a tabbed popup, Along with #82.

a tab control tab for #82
a add/remove/edit custom config tab for this issue

I'd suggest default bootstrap as the design guidelines.

As I can see, this should be Options windows. By clicking icon extension should do:

If current tab is not registered (not active), register this tab and unregister all others.
If current tab is registered (active), unregister itself.

All permanent configuration (like manage services, setup tab control) should be done in options.

makes sense to have the 'what tab gets the control' action as prominent as possible - therefore the icon click itself. the config as an option seems fine, since it's not something a lot of users will do.


Remove Win 386 as OS architecture from OCM CLI binary build

Description
Since there is currently no consumer of the Win 386 binaries, the creation of that OS arch from the OCM CLI build should be removed.
Done Criteria

[ ] Code has been reviewed by other team members
[ ] Analysis of existing tests (Unit and Integration)
[ ] Unit Tests created for new code or existing Unit Tests updated
[ ] Integration Test Suite updated (includes deletion of existing unnecessary Integration Test and/or creation of new ones if required)
[ ] Enduser Documentation updated (if applicable)

kind of related (at least it touches the same source ;-) ) - https://github.com/open-component-model/ocm/issues/947


Health, Guard, & Rpg Item Intergration Request

A few things,
Health:
I was wondering how one goes on about increasing the npc's health, as in #223 it was said that in the config I can do this and also in spigot's yml. Yet issue was locked without mentioning what exactly needed to be changed.
Guard:
When a player tps away from the npc and the chunk unloads, their guard stays put until that chunk is revisited by the player (or another comes along, activates it and it tps back to its owner) Is there a way to keep guard's chunk always active to prevent this?
Rpgitem Request:
Using Rpgitem's item/armor settings with the sentinel
This might relay on citizen's plugin but -shrug- I say that because I'm not sure if it uses citizens inventory.
What I'm wanting to do is equip npcs with Rpg items to give the npc abilities and effects.
Example:
Golden Chestplate
Armour: 25%
Gives speed
Gives off particles when held (or worn)
When I put that in the Sentinel npc at first, it seems to load the armor's protection value (the 25%) so when a player hits it with an rpg sword, they don't easily drill though the npc. (Health drops quicker and I don't want the armor to break) but it doesn't give it speed or show the particles.
When I left and came back, the sentinel started to ignore the armor and it became pretty much a normal golden chestplate.
Inventory / Item Switch:
Something that I just remembered, how do I get an sentinel to switch items? I give it a sword and it won't let me give it a bow without changing the items and when I use the Sentinel inventory command it won't let me add both items.

Health - look through the relevant config files. It shouldn't be hard to find. I just glanced in the Sentinel config and found it very quickly without remembering where it was (it's labeled clearly and has a description info line above it... and there's not that many options that you can't just read through them all).
Guard - There is an option in Citizens config to keep an NPC's chunk loaded. Though also, if it doesn't teleport properly that will need to be handled in a Sentinel code update, as that's a bug.
RPGItem - That's a case of the plugin powering those options needing to account for NPCs intentionally. Can't do anything useful from Sentinel-side short of duplicating that entire plugin within Sentinel. If NBT Attributes are lost on NPC respawn, there might be something failing to save/reload correct in Citizens. Make sure you're running the latest dev build of Citizens and if there's a clear loss of actual on-item data, report that on the Citizens page.
Inventory - You give all the additional items through the inventory (just put items in spare open middle-row-ish slots). It can be a bit glitchy, especially on older versions. Definitely make sure you're up to date there.

Health - Eh I found both health values but the server sees the 2048 value for players only, and the sentinel's health is "what should this npc spawn with?" which works (and is lower than spigot), however it's a one time set, any health value still cannot exceed 2000.
Guard - Thanks, this has been fixed! :)
Rpgitem - Understandable, heh.. just thought it would be cool
Inventory - I figured out a non-glitchy way, add the bow and sword before giving them the trait. My sentinel is now using a bow/sword

any health value still cannot exceed 2000 if that is happening it's not a limit from Sentinel. From what I can see there are no health control calls anywhere that cap at anything other than current max health on the NPC or config granted global max health.
The way you wrote that makes me concerned that you may have edited entirely the wrong value in the Sentinel config. I again urge you to actually read through the provided options. The default spawn health value (of 20 when unmodified) is not the config option to change...

All that I see: https://pastebin.com/dBc0dGiV

Your config is outdated. https://github.com/mcmonkey4eva/Sentinel/blob/master/src/main/resources/config.yml#L62

Thank you! :)

Problem is solved still, just wanted to state: I downloaded the version off of spigot just now. What I had was version 1.4, spigot had 1.4.5.

Problem is solved still, just wanted to state: I downloaded the version off of spigot just now. What I had was version 1.4, spigot had 1.4.5.

Yeah okay, what I failed to do was to regenerate my config file to make sure that the new version is loading that option. Currently for me, 1.4.5 is loading the old config. Thus the new health isn't talking affect.
I feel bad for reopening this.

Join our discord: https://discord.gg/Q6pZGSR

That be great an all if I didn't get locked out of that... I'm waiting on Discord to help me with that since it was 2 stepped and they have to take a month to delete and such. I don't want a new email.
So eh see you guys in a month... :/

We also have an IRC where we can help, it's broadcast to the Discord:
https://one.denizenscript.com/ircchat_denizen

Run latest Sentinel: http://ci.citizensnpcs.co/job/Sentinel/

Okay now it's all fixed. Thanks!


Date is ahead by 1 day when showing events

Its 29th today, right now where I am, 5 am, but the UI is showing 30th Aug, 5 am, 1 day ahead.

Fixed.


Natively support popular external JS libraries

Duplicates

[X] I have searched the existing issues

Latest version

[X] I have tested the latest version

Summary 💡
As discussed here: https://mui-org.slack.com/archives/C02BDJNTKF1/p1666018431708829
we can start by supporting lodash, date-fns.
Should we start by supporting them in functions data source? or everywhere?
Examples 🌈
https://docs.retool.com/docs/custom-js-code
https://docs.appsmith.com/core-concepts/writing-code/ext-libraries
Motivation 🔦
No response

Closing it as it is now supported in custom functions.


Bump up hvpa-controller version to v0.2.0

What this PR does / why we need it:
Bump up hvpa-controller version to v0.2.0:

Introduces new scale mode: "MaintenanceWindow"
Scales limits along with requests

Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Will remove the WIP label once new release of HVPA is created
Release note:

Bump up hvpa-controller version to v0.2.0:
- etcd is scaled down only during maintenance window
- Scales `limits` along with `requests`

hvpa 0.2.0 is release. This PR is ready for the merge.
Thanks to all the reviewers :)


Fixed image paths in docs

Description
I simply fixed the image paths in documentation so that they properly render in the markdown files.
Checklist

[x] I read and understood the contributing guide as well as the code of conduct
[x] My changes generate no new warnings
[ ] My code builds and runs on my machine
[ ] I documented my code
[ ] Review requested

Screenshots

Since DocC automatically resolves resources in the Resources folder inside the catalog, this is not needed. Please do some research first and try to build the documentation in Xcode first before suggesting changes.
Swift DocC Documentation (Images)

https://developer.apple.com/documentation/xcode/formatting-your-documentation-content#Add-images-to-your-content


Autofocus on password field

Save ourselves one click / tab press

@maksimf Thanks for the contribution!
Any opinion on this @danrabinowitz? In the past I didn’t like it because of how iOS “zoomed” the page in on auto-focused fields, but I don’t think that behavior exists anymore, so I’m more open to implementing it.

I like the idea. 👍

I'm traveling this week, and have poor network access, so I can test it myself until this weekend.

Thanks gents! I just pushed 1.4.5 with this update.


Suggestion window is empty

After selecting the layers and pressing Ctl+Shift+M, the window is not showing any suggestions.

Can you access https://inputtools.google.com/request?ime=handwriting&app=autodraw&dbg=1&cs=1&oe=UTF-8 with your browser? It should return a text file with error description.

[
"FAILED_TO_PARSE_REQUEST_BODY",
[],
"",
{
"bns" : "/bns/sa/borg/sa/bns/i18n-input-prod/input-frontend.ife/7",
"engine_info" : [],
"experiment_id" : [],
"is_valid_request_data" : false,
"original_input_method_name" : "handwriting"
}
]

same here.
sketch 50.2


Fix the CI loop

Looks like the virtual environments broke the CI loop.
asklar did some digging already, he said this about Chris's PR which hit the build error:
Going back to the original question – I see you hit this error @Chris Guzak (WINDOWS)
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\DesktopBridge\Microsoft.DesktopBridge.targets(16,3): error MSB4019: The imported project "C:\Users\runneradmin\AppData\Local\Microsoft\dotnet\sdk\3.1.201\Sdks\Microsoft.NET.Sdk\Sdk\Sdk.NuGet.targets" was not found. Confirm that the expression in the Import declaration "C:\Users\runneradmin\AppData\Local\Microsoft\dotnet\sdk\3.1.201\Sdks\Microsoft.NET.Sdk\Sdk\Sdk.NuGet.targets" is correct, and that the file exists on disk. [D:\a\Xaml-Islands-Samples\Xaml-Islands-Samples\Samples\Win32\SampleCppApp.Package\SampleCppApp.Package.wapproj]
Where you able to figure out what’s causing it? I checked the virtual environments page and they do have .net 3.1.201 installed; however it is not to the user’s folder as the error above expects it to be, but to Program files:
virtual-environments/Windows2019-Readme.md at main · actions/virtual-environments (github.com)
Location C:\Program Files\dotnet\sdk
So I think this is a case where the virtual environments made a change that broke CI.  I filed https://github.com/actions/virtual-environments/issues/3704 to track.

The GitHub actions folks replied to https://github.com/actions/virtual-environments/issues/3704, saying they will investigate, but also suggested I switch the CI loop to use https://github.com/actions/setup-dotnet
miketimofeev says:
I'm also asking this because the latest patch version for 3.1.2xx is 3.1.202 and we will probably remove all the versions except the latest patch in the future as we did for Ubuntu #3338
@asklar do you think it makes sense to switch over to dotnet build as they are suggesting?

I'm not sure how that would help since the project is native C++? the WAP is the thing that is kind of enmeshed with .net. You could try the dotnet thing but it doesn't sounds like it would help, worth trying though :)

I temporarily disabled package signing to unblock the CI loop.
Filed #61 to track re-enabling package signing.


Bad perpomance on nerfstudio dataset (kitchen dataset) using mipnerf and vanilla-nerf.

After 2 days of training, I still get bad rendering result.

mipnerf isn't set up to work well with real scenes. It is optimized for blender scenes. If you want to reconstruct real-scenes, we reccomend using the nerfacto model.


JWST NIRSPEC spectrum not plotting

https://simple-bd-archive.org/load_solo/CWISEP J193518.58-154620.3.
The spectrum is there, I don't see an error being thrown, but I don't see any spectrum being displayed.

Okay, the spectrum is being plotted but because it's 2.86--5.14um, it's not immediately apparent. Looks like a bug in my plotting code of assuming SpeX-like spectra.


Ability to select from alternate titles (& Different UI for them)

Describe your suggested feature
I am thinking of something like neko (Tachi Fork) offers.
Show alternate in similar UI to tags & ability to replace anyone of them with the title.

Other details
No response
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] I have updated the app to version 0.16.4.
[X] I will fill out all of the requested information in this form.

Seems like basically a duplicate of #5 which would get fixed by #398

Ok It is same as them.

I don't think it is a duplicate. Allowing sources to provide alternative titles removes the manual work of editing manga info per title.
The replacement option does get fixed by edit manga info, but it feels like there is room for an improvement.
I do know extension API changes are all currently on hold though, so I wouldn't expect it as a feature any time soon.


Move from Docker to UCR for Confluent Replicator

All other Confluent components use the UCR instead of Docker. Setting this to UCR allows us to monitor the memory usage for example too. Tested and verified on our test and production clusters.

thanks @wjglerum!

Thanks for the PR @wjglerum ! I tested the MESOS container type and it was not working when  virtual network was not enabled because of the parameter network: "BRIDGE". I will do some more changes and will cherry-pick your commit.

Thanks for the feedback. We are running with MESOS and BRIDGE mode and that seems to work. What's going wrong?

With container changed to Mesos and keeping things as is in a non virtual neteork enabled scenario, I was getting
/container/docker/network should be empty and /container/docker/network should be empty

So I made the changes similar to that we have in other confluent packages here
Can you check and let me know if these changes work for you also?

Makes sense, I only edited the generated Marathon JSON directly, so I haven't used the templating from the package. Thanks for the info!


Addon Name

Addon Manifest URL
https://wospor.surge.sh/manifest.json
Addon Description
No response
Language of Content
No response
Choose Labels

[ ] anime
[ ] asian drama
[ ] bollywood
[ ] cartoons
[ ] debrid support
[ ] http streams
[ ] live tv
[ ] metadata
[ ] misc
[ ] movies
[ ] music
[ ] nsfw
[ ] podcasts
[ ] radios
[ ] subtitles
[ ] torrents
[ ] tv shows
[ ] video games

i don't know what that is, but it's not a new addon, so i'm closing this


Differentiate between failed thumbnail url and emtpy url images.

#1808
Hi, I would need an icon image to be inserted on this issue. Code is ready to be reviewed but I'm missing a second icon, I'm very bad at design, so if anyone can chime in, thanks a lot!
On Failed url:

On Emtpy url:

@NetOperatorWibby, think you can give us a hand?

@tzarebczan How do I add images to an existing PR?

@dan1d where is the image pulling from? I don't see it in the PR.
@NetOperatorWibby since this is a community PR, the branch is on his forked repo. I'd assume you can edit there, or just add the thumbnail to this thread.

@dan1d we'd probably want to define the warning thumbnail the same way, right?
@NetOperatorWibby might just be easier to include a png here (it would be the same size/aspect ratio as the one we provided for the no thumb scenario: https://github.com/lbryio/lbry-desktop/pull/1755/files

This should work.

This should work.

Looks good!

Above could probably be done in pure CSS. More importantly, it should probably be the error color.

+1 for error color but I went with image since the other message is also an image. Is there a unified icon set? It would make sense to take the CSS approach then.

Used the error color found here.

Image updated to red, but should we add them with CSS then ?


Check if upgrade Sphinx to 3.1.2 causes any issues

Installing Sphinx-js 3.0 requires higher version of Sphinx.

Looks like everything compiles fine and without errors, so I'm going to close this


--release

You should write to use --release in the README.

README now suggests cargo install, which defaults to --release.


Numbers not allowed in vendor names ?

Problem
While learning to develop extension, I have tried a little widget extension which worked fine under its original vendor name.
So I have modified it under my own vendor name which was f1evm : and it did not worked.
In the parameters folder, when adding the widget, I only had the input field for the Title instead of my settings.
So I modified my vendor name : f1evm  -->  fevm  (I deleted the number for the packages/fevm folder name and inside the files).
And then, it worked again.
Does this mean that numbers are not allowed for the vendor name ?
testwidget.zip
Technical Details

Pagekit version: 1.0.12
Webserver: Apache
Database: mysql 10.1.23-MariaDB
PHP Version: 5.6.31

Hello,
I do not know, whether this is a bug or not.
I did not see packages with numbers in their vendor names yet. In the documentation you can find:

Note: If you don't know what to use as a vendor name, your GitHub username is usually a good bet. While package names are case insensitive, the convention is all lowercase and dashes for word separation.

(https://getcomposer.org/doc/02-libraries.md#every-project-is-a-package)
So the convention is lowercase and words separated by dash - but it does not explicitly exclude numbers?


Create a new pull request by comparing changes across two branches

Description

Motivation & Context

Type of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Refactoring (non-breaking change)
[ ] Breaking change (fix or feature that would change existing functionality)

Checklist

[ ] I have read the contributing guidelines
[ ] My code follows the code style of the project (using npm run lint)
[ ] My change introduces changes to the documentation
[ ] I have updated the documentation accordingly
[ ] I have added tests to cover my changes
[ ] All new and existing tests passed

Live previews

https://deploy-preview-{your_pr_number}--twbs-bootstrap.netlify.app/

Related issues

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-0.06%) to 95.717%

Files with Coverage Reduction
New Missed Lines
%

js/src/util/focustrap.js
1
91.67%

Totals

Change from base Build<PHONE_NUMBER>:
-0.06%

Covered Lines:
2021

Relevant Lines:
2072

💛  - Coveralls