Notes after initial uploads to repository

Submitter: Rob Crystal-Ornelas
I suggest the following changes:
After reading through the associated manuscript, and adding content to the GitHub repo, there are a few other additions we should consider

[ ] Which usage license should we select. In ESS-DIVE you’ll have the option to select CC BY 1.0 or CC BY 4.0. Then, I can create a .txt file for the license so that is clearly visible in the GitHub repo.

[ ] Which document should we encourage users to cite on our README page? The manuscript that is soon to be submitted?

[ ] I created a small set of GitHub issue templates that can help to organize feedback. Any other templates that I should add or are the 2 that I have OK for now?

Notes from our meeting today:
I'll take care of all updates, except for one point below.

No need for data package to be submitted to ESS-DIVE since the GitHub Page will post summary of manuscript
We'll indicate a CC BY 4.0 license
Add citation for manuscript to first paragraph of README
Change headings, and summarize how to use each figure in instructions with a bullet point list
@msimmond will update figures (to remove words that reference parts of the manuscript), then let Rob know when they'r ready to go.
Move figure footnote so that it's written in markdown text and links are active.

@msimmond I think we still need to update the FLMD figure, is that right? After that, I think those are the last of the major content edits 🙌

@robcrystalornelas yes, it's updated now in the google slides - I removed the reference to a figure. Thanks!

Great, thanks! I'll get the figure updated today.
On Tue, Apr 27, 2021 at 12:29 PM Maegen Simmonds @.***>
wrote:

@robcrystalornelas https://github.com/robcrystalornelas yes, it's
updated now in the google slides - I removed the reference to a figure.
Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ess-dive-community/essdive-model-data-archiving-guidelines/issues/4#issuecomment-827860850,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADTX7SNIKXX3W64NBJE37WDTK4F7XANCNFSM43JCCBAQ
.

--
Rob Crystal-Ornelas, PhD
Postdoctoral Scholar
Lawrence Berkeley National Lab | ESS-DIVE
Pronouns: he/him/his

Finished uploading new FLMD image, and fixed links


TFRS - Enhance Messaging to Sync Compliance Reports from TFRS to LCFS

Description
This enhancement introduces a messaging-based synchronization mechanism ensuring that compliance reports created or updated in TFRS are immediately known to LCFS. Previously, LCFS only received messages related to transactions upon a compliance report reaching certain states (e.g., Director approval). With this change, when a compliance report is created in TFRS, a corresponding message is published to a queue that LCFS consumes. LCFS then creates a local compliance_report entry in advance, enabling seamless handling of subsequent events, such as placing credits "in reserve" or linking approved credit trades.
Purpose and Benefit
By proactively sharing compliance report information from TFRS to LCFS:

LCFS maintains an up-to-date reference of all compliance reports.
When a report transitions to a submitted or approved state in TFRS, LCFS can immediately attach transactions or credits to the correct compliance_report record.
Reduces complexity and ensures synchronous data integrity between TFRS and LCFS.

Acceptance Criteria

Creation Message: When a compliance report is created in TFRS, a message is published to a designated compliance_report_queue with essential details (tfrs_compliance_report_id, organization_id, compliance_period_id, report_type, nickname).
LCFS Record Creation: Upon receiving the message, LCFS creates a corresponding compliance_report record with a compliance_report_group_uuid and version=0 if it represents the original report.
Subsequent Updates: When TFRS later submits or approves the compliance report, LCFS already has the reference and can create "In reserve" transactions or other compliance-unit related changes without delay.
Error Handling: Failures in message publishing or consumption are logged. The system continues to process other messages.
Documentation & Schema: The message schema and code usage are documented.

## Message Schema Example
{
 "tfrs_compliance_report_id": 1234,
 "organization_id": 567,
 "compliance_period_id": 89,
 "report_type": "Compliance Report",
 "nickname": "Initial Report",
 "timestamp": "2024-01-15T12:34:56Z"
}

Development Checklist

Message Schema Definition:
Define a JSON schema for compliance_report messages, including fields for TFRS report ID, organization, compliance period, report type, and nickname.

Message Publishing in TFRS:
In ComplianceReportService (TFRS side), after creating a compliance report, publish a message:
from api.services.ComplianceReportMessageService import ComplianceReportMessageService

   class ComplianceReportService:
       @staticmethod
       def create_compliance_report(organization_id, compliance_period_id, report_type, nickname=None):
           compliance_report = ComplianceReport.objects.create(
               organization_id=organization_id,
               compliance_period_id=compliance_period_id,
               type=ComplianceReportType.objects.get(the_type=report_type),
               nickname=nickname
           )
           ComplianceReportMessageService.send_compliance_report_message(
               tfrs_id=compliance_report.id,
               organization_id=organization_id,
               compliance_period_id=compliance_period_id,
               report_type=report_type,
               nickname=nickname
           )
           return compliance_report

Message Consumption in LCFS:
In LCFS, implement a consumer that listens to compliance_report_queue:

   import json, uuid
   from lcfs.db.models import ComplianceReport
   def on_compliance_report_message(ch, method, properties, body):
       message = json.loads(body)
       tfrs_id = message["tfrs_compliance_report_id"]
       org_id = message["organization_id"]
       period_id = message["compliance_period_id"]
       report_type = message["report_type"]
       nickname = message.get("nickname")
       lcfs_report = ComplianceReport(
           compliance_period_id=period_id,
           organization_id=org_id,
           compliance_report_group_uuid=str(uuid.uuid4()),
           version=0,
           nickname=nickname
       )
       lcfs_report.save()
       print(f"Created LCFS compliance_report for TFRS ID {tfrs_id}")

Transaction Integration:
With LCFS aware of compliance_reports from their inception, "In reserve" transactions or other updates can be applied immediately upon status changes.

Error Handling and Logging:
Implement try-except around message operations and log failures. Retry or queue dead-letter logic as needed.

Testing:

Create a compliance report in TFRS and verify a message is published.
Ensure LCFS consumes the message and creates a compliance_report entry.
Submit and approve the compliance report in TFRS; verify LCFS can create "In reserve" transactions.
Test edge cases (missing fields, network issues).

Documentation:
Document message flows, queue configurations, and code integration. Include diagrams and examples of normal and exceptional flows.

Notes
This approach ensures LCFS has prior knowledge of compliance reports referenced by transactions or approvals in TFRS. By maintaining a synchronized state, both systems can operate efficiently, providing a more reliable and consistent compliance management experience.

Talked to Alex:
We are going to be adding legacy_id to compliance reports in LCFS to indicate which ones are from TFRS to control certain behaviour. This also keeps us from needing to worry about auto-increment IDs clashing on each system.

Sync is working on Dev!


Collision Warning persists at Zero Speed

콤마3 이슈 외에 제보를 받고있지 않습니다.
please post only comma3 issues.
*Describe the bug
Low speed closure on a large vehicle (American van, American pickup truck, medium to heavt duty truck - a vehicle that fills most (>~90%) of the Comma3's road display) generates Red screen on Comma3 collision warning ("Brake! Risk of Collision") which persists despite the Bolt EV having reached zero speed through regen (one pedal driving - do not use brake pedal).
To Reproduce
Steps to reproduce the behavior:

Go to within a meter of stopped truck ahead while decelerating with regen paddle only (one pedal driving, do not tough brake pedal)
Observe collision alert activate & persist
Slow to zero speed with regen paddle only with target of a half meter follow.
See error where alert screen persists for up to infinite time (very annoying audio and red screen) until either vehicle ahead moves forward to reduce its size in the Comma3 screen or (I think) if brake pedal cancel (I don't believe the regen pasdle can cancel the alerter).

Expected behavior
Vehicle alerts on leading truck's image mostly filling Comma3 screen during one prdal driving. Irrespective of leading truck's image size in Comma3 screen, collision alert cancels when Bolt EV regen brake achieves zero speed or when truck image size on screen gets smaller (Bolt reversing or truck moving forward).
Additional context
none

please tell me this bug occured in branch "0819" ?

yes
Sent from my iPhone

On Jul 26, 2023, at 1:26 AM, jc01rho @.***> wrote:

please tell me this bug occured in branch "0819" ?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.

collision warning has combind Commas video read model. (model_fcw, planner_fcw)
https://github.com/jc01rho-openpilot-BoltEV2019-KoKr/boltpilot/blob/4701b1aaa5dd189f2cf2bd9ccba5507e249142fc/selfdrive/controls/controlsd.py#L426-L426
for now, I'd recommend disable collision warning.
find "Disable Openpilot FCW" and toggle off :)


[ISSUE] When zeebeGateway.ingress.rest.enabled contextPath and paths are causing issues.

Describe the issue:
When  zeebeGateway.ingress.rest.enabled is set to true we have constraints that the zeebeGateway.contextPath and zeebeGateway.ingress.rest.path need to be the same.

Actual behavior:
I then set zeebeGateway.contextPath= / and zeebeGateway.ingress.rest.path= /
zeebe-gateway:
  replicas: 1
  affinity: {}
  contextPath: /
  ingress:
    rest:
      enabled: true
      host: zeebe-gateway.dev.cosmdream.com
      tls:
        enabled: true
        secretName: "dev-cosmdream-com-tls"
      path: /

when running it the zeebe-gateway pod doesn't get in the ready state. Looking at the deployment we can see there is an additional / in the readiness probe.
        readinessProbe:
          failureThreshold: 5
          httpGet:
            path: //actuator/health/readiness

Then I edited and removed the /
        readinessProbe:
          failureThreshold: 5
          httpGet:
            path: /actuator/health/readiness

After that you need to kill the pod and confirm the correct one runs.

Expected behavior:
There should be no extra / and shouldn't have to edit the deployment to get a working configuration.

How to reproduce:

Logs:

Related:
SUPPORT-23427
Please note: Without the following info, it's hard to resolve the issue and probably it will be closed.

Platform:
Helm CLI version:
Chart version:
Values file:

Since you have added the constraint that when zeebeGateway.ingress.rest.enabled is set to true the zeebeGateway.contextPath and zeebeGateway.ingress.rest.path need to be the same, when you fix this, you should also fix it to set the default value of zeebeGateway.ingress.rest.path: \.  It is a bit strange that I do not set zeebeGateway.contextPath in my values.yaml, so it has the default value "/", so now I have to add to my values.yaml a new value that I never needed before: zeebeGateway.ingress.rest.path: \, just to satisfy a constraint that was added.  The constraint should already be satisfied in the default values.yaml.

To conclude, double slashes // in URI paths are not sanitized to single slashes / by spring. You may be able to configure that overriding some defaults but that may cause issues on spring security which apparently is not supporting sanitizing either. I wouldn’t go for that due to the risk of side-effects down the line.
See also https://github.com/spring-projects/spring-framework/issues/15845#issuecomment-453407951. They also mention that according to RFC3986 double slash characters are actually illegal see here.


Dotcom 107543 fix region nav

fix for columbian link
Resolves: Dotcom-107543
Test URLs:

Before: https://stage--express--adobecom.hlx.page/express/?martech=off
After: https://dotcom-107543-fix-region-nav--express--adobecom.hlx.page/express/?martech=off

@vhargrave Still trying to figure out how to review this but we seem to have failed a test:


Level loader code has no error checking

Loader has been written very fast just to test the graphics engine, so it has no error checking/reporting.

IFF syntax errors handling as well as numerous data checks have been added in commit 875ea93.

IFF syntax errors handling as well as numerous data checks have been added in commit https://github.com/grzegorz-kraszewski/untangle/commit/875ea939e758d58e71cdb75324cd1a23d5352976.


Improved isUnsigned to check attribute minimum more carefully

The previous implementation checked the validation predicate for a string containing literally ">= 0".  But if an attribute's minimum was, for example, 1 instead of 0, the test would fail because the string would be ">= 1".  But when a minimum is 1 we still want to use unsigned, but weren't getting unsigned because of this.
Updated the code to instead look for the string "SELF >=" and then inspect the number that comes after (if any). Then we can in fact correctly check that the minimus is >= 0, thus allowing unsigned to be used in those cases too.

Thanks @rentzsch !
After all these years, we are updating from mogen 1.27. :)


Support non-square 2D grids

For simplification, the current implementation only supports square grids. For broader use, it would be useful to extend support to rectangular grids.

It doesn't crash when sorting non-square grids, but the code in the grid reorganization has only one variable for block size and number of blocks (independent of x/y dim), e.g. https://github.com/fraunhoferhhi/PLAS/blob/main/plas.py#L238-L239 - thus it doesn't properly sort the non-square overhanging part.
Doesn't really make sense fixing this state of the code for non-square grids, as it contains the overly complicated but dead/unused pixel unshuffling.
-> do non-square support after rewrite


ACF w/o SIMD

We have GPU and CPU options for ACF computation.  The shader based GPU implementation may need more work to achieve desired accuracy.  For the CPU path there is NEON and SSE support (thanks to NEONvsSSE.h header).  Some of this code will currently no run on the CPU without SIMD.  Boost.SIMD might be worth a look to generalize some of that code.

Moved: https://github.com/elucideye/acf/issues/18


Console localization fix

Added localization support for the EDC sample.
Added a RESW file with an example of fields that should be translated.
Modified readme to include more information about localization.

@timch-ms If you could take a quick glance at this PR for localization support, I would appreciate it.

Looks good. I would suggest though, you put in another language in the template and ensure you test it to make sure it works. Maybe work with one of our Japan teammates to get a translation, though i think for what you have a Bing translate is sufficient.


Add input_datetime reload service.

Description:
Add input_datetime.reload admin service so new entities could be added without entire HA instance restart
Checklist:

[x] The code change is tested and works locally.
[x] Local tests pass with tox. Your PR cannot be merged unless tests pass
[x] There is no commented out code in this PR.
[x] I have followed the development checklist

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated in home-assistant.io

If the code communicates with devices, web services, or third-party tools:

[ ] The manifest file has all fields filled out correctly. Update and include derived files by running python3 -m script.hassfest.
[ ] New or updated dependencies have been added to requirements_all.txt by running python3 -m script.gen_requirements_all.
[ ] Untested files have been added to .coveragerc.

If the code does not interact with devices:

[x] Tests have been added to verify that the new code works.

@Adminiuga Documentation for this is missing. Could you please open up a PR in our documentation repository?


Getting error using sqlite on arch

I install the sqlite package with pacman, have the setup as it should be. I get the error:
Found sqlite.lua: but got the following error: module 'sqlite' not found:
        no field package.preload['sqlite']
lazy_loader: module sqlite not found
lazy_loader_lib: module sqlite not found
        no file './sqlite.lua'
        no file '/usr/share/luajit-2.1.0-beta3/sqlite.lua'
        no file '/usr/local/share/lua/5.1/sqlite.lua'
        no file '/usr/local/share/lua/5.1/sqlite/init.lua'
        no file '/usr/share/lua/5.1/sqlite.lua'
        no file '/usr/share/lua/5.1/sqlite/init.lua'
        no file './sqlite.so'
        no file '/usr/local/lib/lua/5.1/sqlite.so'
        no file '/usr/lib/lua/5.1/sqlite.so'
        no file '/usr/local/lib/lua/5.1/loadall.so'
Error detected while processing VimEnter Autocommands for "*":
Error executing lua callback: .../share/nvim/lazy/yanky.nvim/lua/yanky/storage/memory.lua:12: attempt to index field 'config' (a nil value)
stack traceback:
        .../share/nvim/lazy/yanky.nvim/lua/yanky/storage/memory.lua:12: in function 'push'
        .../.local/share/nvim/lazy/yanky.nvim/lua/yanky/history.lua:21: in function 'push'
        ...e/jordan/.local/share/nvim/lazy/yanky.nvim/lua/yanky.lua:59: in function <...e/jordan/.local/share/nvim/lazy/yanky.nvim/lua/yanky.lua:58>
Press ENTER or type command to continue

nevermind, I made a mistake


Include creators and translators names in the scenarios

Everyone should have the chance to put his name/nickname in the scenario and when someone sees a scenario can see who created it and who translated some language.
I suggest to add a authors element in the quest properties screen.
In the translation file, originalLanguage will have the creators names and in the foreign languages will be the translators names.
ie
//Lynch
quest.authors,NPBruce,Eorahil,,[GermanTranslator],,,,,,

I envisaged this being meta data on the quest hosting system that doesn't exist yet.  Either way something should be done.


Custom MSBuild properties are not resolved by R#

It is possible to define custom build properties in .targets files or in Visual Studio project files, for example:
<PropertyGroup>
    <TemplatesDir>$(SolutionDir)Templates\</TemplatesDir>
</PropertyGroup>

When transforming templates from Visual Studio or in the build, these properties are recognized and substituted correctly.
However, when ForTea is added to ReSharper, and user-defined properties are used in include or assembly T4 directives, they are underlined with red and on mouseover, an error is displayed saying that the file could not be found (and in the path, the variable is unresolved, just the $() is stripped so e.g. $(TemplatesDir)template.ttinclude becomes TemplatesDirtemplate.ttinclude).
The actual transformation still succeeds, so it's just a false error report from ForTea, which also breaks usage references and code navigation.
If you would consider this something to be fixed, I will make a pull request based on our changes.

Nice catch! I'll happily accept a pull request.


data mapping for pointer fields will overwrite every item in the slice

sq's multi-row struct mapping works by repeatedly binding data to a single variable, then appending that variable to a slice. This works if the variable is a value type, since it will be copied by value every append: subsequent modifications to that variable will not affect what has already been appended into the slice. However if there were any fields that are pointer types, subsequent modifications to those fields will overwrite them for everything that has already been appended to the slice (since they were copied by reference).
There are workarounds for this (manually copying data before append), but it's not very user friendly as it requires understanding the inner workings of the library in order to fully grok.
I'm honestly not sure if there's a way around it, since the current way is the only way to have a fully 'generic' mapper function that works for any data type. By encapsulating stateful modifications in a closure function (or a pointer receiver method), the function for all mapper functions can have the same type of func (*sq.Row).
Even Go generics may not be sufficient here as they do not permit type parameters on methods. The following code below is ideal, but will not be achievable with Go's generics:
func (q SelectQuery) Get[Item any](db *DB, mapper func(*Row) Item) (Item, error)
func (q SelectQuery) GetSlice[Item any](db *DB, mapper func(*Row) Item) ([]Item, error)
func (q SelectQuery) GetAccumulate[Item, Items any](db *DB, initial Items, mapper func(*Row) Item, accumulator func(Item, Items) Items) (Items, error)

The document states that 'a suitably parameterized top-level function' will have to be written. Perhaps that may be the workaround.
func Get[Item any](db *DB, q Query, mapper func(*Row) Item) (Item, error)
func GetSlice[Item any](db *DB, q Query, mapper func(*Row) Item) ([]Item, error)

However the invocation will be so clunky compared to regular Selectx that I am not so sure that it is a direct upgrade:
// using Selectx
var user User
var user []User
err := sq.
    From(u).
    Where(u.Name.EqString("bob")).
    Selectx(user.RowMapper, func() { users = append(users, user) }).
    Fetch(db)
if err != nil {
}
fmt.Println(users)

// using GetSlice
users, err := sq.GetSlice(db,
    sq.From(u).
        Where(u.NAME.EqString("bob")),
    User{}.RowMapper,
if err != nil {
}
fmt.Println(users)
)

The only value Get brings over Selectx is that the mapper function can return a new value everytime rather than doing some side effect on a pointer receiver. My main issue with this is that it solves the problem of overwriting pointers fields but with an entirely different way of doing things (converting the query from a method to a top level function).

Why not use this use-case as an argument for permitting type parameters on methods?

Nah I don't think it's possible to convince the Go team to budge on this, I think they have some performance issues with it. Besides once generics roll out on Feb 2022, more people will be trying it out and if type parameters on methods prove to be a paint point for many people it might get added eventually.

@bokwoon95 fair enough, I guess. Let's hope it doesn't get delayed.


Several Snapshtot and Drain Fixes

feat(snapshot): add runtime snapshot list to volume

Signed-off-by: Tiago Castro<EMAIL_ADDRESS>

fix(drain): notify reconciler when a node drain command is received

Signed-off-by: Tiago Castro<EMAIL_ADDRESS>

fix(drain-cordon): missing pstor flush after refactoring

This would cause drain and cordon operations not to log writes to pstor.

Signed-off-by: Tiago Castro<EMAIL_ADDRESS>

test(snapshot/timeouts): enhance snapshot create timeout testing

Signed-off-by: Tiago Castro<EMAIL_ADDRESS>

fix(snapshot/create): handle create errors

When snapshot creation fail, attempt to undo it to pstor. This can happen when creating snapshot
through nexus and nexus crashes but replica snapshot is created.
Potentially we could accept this scenario as a success, but for now let's start on the safe side.

Signed-off-by: Tiago Castro<EMAIL_ADDRESS>

refactor(snapshot/cleanup): destroy transactions

Rework transaction cleanup to wa potential order issue.

Signed-off-by: Tiago Castro<EMAIL_ADDRESS>

bors try

bors merge

bors merge


fix: Wrong Scala version in Databricks integration tests configuration.

Problem
The configuration for Databricks integration tests includes Java 2.13. Databricks supports only Scala 2.12
Solution
We should replace faulty Scala version

[ ] Your change modifies the core OpenLineage model
[ ] Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).
One-line summary:
Checklist

[X] You've signed-off your work
[X] Your pull request title follows our guidelines
[ ] Your changes are accompanied by tests (if relevant)
[X] Your change contains a small diff and is self-contained
[ ] You've updated any relevant documentation (if relevant)
[ ] Your comment includes a one-liner for the changelog about the specific purpose of the change (not required for changes to tests, docs, or CI config)
[ ] You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
[ ] You've added a header to source files (if relevant)

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2024 contributors to the OpenLineage project

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 85.75%. Comparing base (0399867) to head (eef119b).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@             Coverage Diff              @@
##              main    #3236       +/-   ##
============================================
- Coverage   100.00%   85.75%   -14.25%
============================================
  Files            5       54       +49
  Lines          173     3124     +2951
============================================
+ Hits           173     2679     +2506
- Misses           0      445      +445

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Selected option's text isn't localized

In the users view, the selected option of the role-selector is never localized after page load.

I'm working on a fix for this.

Fixed in https://github.com/llaske/sugarizer-server/commit/ac791858f1f0bb968e280944d36eb235b3a628c4


Less generation when no messages, enums, or extensions.

There's no need to generate any import directives or the version check when the file is completely empty, and with different access levels on imports that can get flagged with a newer compilers.
This is a common case when a .proto file contains a service.
Regenerate to get the one file update.

@gjcairo fyi


SwiftReorder blocks other UIGestureRecognizers in tableview cells

I have a UITableView with custom cells.  In each cell I have a thumbnail image and each thumbnail has a UIGestureRecognizer which responds to long presses and triggers a full screen overlay with a zoomed version of the thumbnail.
After implementing SwiftReorder, my cells are no longer responding to the long press on thumbnail images.  Is there a way to have SwiftReorder ignore gestures if they happen on a thumbnail image and send those gestures to my pre-exiting UIGestureRecognizer instead?

Update: I'm sure there is a better way to achieve this, but I was able to get the functionality back by ensuring the minimumPressDuration on my thumbnail image gesture recognizer was set to a lower value than the tableView.reorder.longPressDuration property for SwiftReorder.

You can use the TableViewReorderDelegate.
Implement the canReorderRowAt delegate method, and others as necessary.
func tableView(_ tableView: UITableView, canReorderRowAt indexPath: IndexPath) -> Bool {         //If it is a thumbnail cell, then return false     }


chore: package exports add types

At nuxt.config.ts, it reslove type schema from config item exports types by the tsconfig "moduleResolution": "Bundler"

Thank you!


Cairo build fails on OS X El Capitan

Cairo.jl encounters build errors during Pkg.add("Cairo") on julia 0.3.11 as well as 0.4.0-rc3.
==> ./configure --disable-silent-rules --prefix=/Users/datafig/.julia/v0.4/Homebrew/deps/usr/Cellar/pixman/0.32.6 --disable-gtk
==> make install
Last 15 lines from /Users/datafig/Library/Logs/Homebrew/pixman/02.make:
/bin/sh ../libtool  --tag=CC   --mode=compile clang -DHAVE_CONFIG_H -I. -I..    -mmmx -Winline -g -O2 -Wall -Wdeclaration-after-statement -fno-strict-aliasing -fvisibility=hidden -c -o libpixman_mmx_la-pixman-mmx.lo `test -f 'pixman-mmx.c' || echo './'`pixman-mmx.c
/bin/sh ../libtool  --tag=CC   --mode=compile clang -DHAVE_CONFIG_H -I. -I..    -msse2 -Winline -g -O2 -Wall -Wdeclaration-after-statement -fno-strict-aliasing -fvisibility=hidden -c -o libpixman_sse2_la-pixman-sse2.lo `test -f 'pixman-sse2.c' || echo './'`pixman-sse2.c
libtool: compile:  clang -DHAVE_CONFIG_H -I. -I.. -mmmx -Winline -g -O2 -Wall -Wdeclaration-after-statement -fno-strict-aliasing -fvisibility=hidden -c pixman-mmx.c  -fno-common -DPIC -o .libs/libpixman_mmx_la-pixman-mmx.o
libtool: compile:  clang -DHAVE_CONFIG_H -I. -I.. -msse2 -Winline -g -O2 -Wall -Wdeclaration-after-statement -fno-strict-aliasing -fvisibility=hidden -c pixman-sse2.c  -fno-common -DPIC -o .libs/libpixman_sse2_la-pixman-sse2.o
/bin/sh ../libtool  --tag=CC   --mode=compile clang -DHAVE_CONFIG_H -I. -I..    -mssse3 -Winline -g -O2 -Wall -Wdeclaration-after-statement -fno-strict-aliasing -fvisibility=hidden -c -o libpixman_ssse3_la-pixman-ssse3.lo `test -f 'pixman-ssse3.c' || echo './'`pixman-ssse3.c
libtool: compile:  clang -DHAVE_CONFIG_H -I. -I.. -mssse3 -Winline -g -O2 -Wall -Wdeclaration-after-statement -fno-strict-aliasing -fvisibility=hidden -c pixman-ssse3.c  -fno-common -DPIC -o .libs/libpixman_ssse3_la-pixman-ssse3.o
pixman-mmx.c:100:20: error: constraint 'K' expects an integer constant expression
        : "y" (__A), "K" (__N)
                          ^~~
1 error generated.
make[1]: *** [libpixman_mmx_la-pixman-mmx.lo] Error 1
make[1]: *** Waiting for unfinished jobs....
libtool: compile:  clang -DHAVE_CONFIG_H -I. -I.. -mssse3 -Winline -g -O2 -Wall -Wdeclaration-after-statement -fno-strict-aliasing -fvisibility=hidden -c pixman-ssse3.c -o libpixman_ssse3_la-pixman-ssse3.o >/dev/null 2>&1
libtool: compile:  clang -DHAVE_CONFIG_H -I. -I.. -msse2 -Winline -g -O2 -Wall -Wdeclaration-after-statement -fno-strict-aliasing -fvisibility=hidden -c pixman-sse2.c -o libpixman_sse2_la-pixman-sse2.o >/dev/null 2>&1
make: *** [install-recursive] Error 1

READ THIS: https://git.io/brew-troubleshooting
If reporting this issue please do so at (not Homebrew/homebrew):
  https://github.com/staticfloat/homebrew-juliadeps/issues

Warning:
==================================================================[ ERROR: Cairo ]===================================================================

LoadError: failed process: Process(`/Users/datafig/.julia/v0.4/Homebrew/deps/usr/bin/brew install --force-bottle staticfloat/juliadeps/cairo`, ProcessExited(1)) [1]
while loading /Users/datafig/.julia/v0.4/Cairo/deps/build.jl, in expression starting on line 144

=====================================================================================================================================================

==================================================================[ BUILD ERRORS ]===================================================================

WARNING: Cairo had build errors.

 - packages with build errors remain installed in /Users/datafig/.julia/v0.4
 - build the package(s) and all dependencies with `Pkg.build("Cairo")`
 - build a single package by running its `deps/build.jl` script

=====================================================================================================================================================
INFO: Package database updated

I was able to build Cairo successfully by using a hack. During installation the process creates a Homebrew formula named pixman.rb under $HOME/.julia/v0.4/Homebrew/deps/usr/Library/Taps/staticfloat/homebrew-juliadeps. I don't know what script is responsible for generating the formula. But if the parameter --disable-mmx is added to it right after the file is generated the compile concludes successfully.
A more reliable fix would be to have the modification made on the script that is responsible for generating the formula. But for now i was able to make it work.

+1

I still have problems installing Cairo.jl on OS X 10.11.1. The build fails. The --disable.mmx flag is set in the pixman.rb file but doesn't seem to help.
julia> Pkg.build("Cairo")
INFO: Building Homebrew
HEAD is now at cd49526 wireshark 2.0.0rc3 (devel)
HEAD is now at b178725 Merge pull request #83 from staticfloat/staging
INFO: Building Cairo
==> Installing glib from staticfloat/homebrew-juliadeps
==> Downloading https://s3.amazonaws.com/juliacache/glib-2.44.1.tar.xz
Already downloaded: /Users/feldt/Library/Caches/Homebrew.jl/glib-2.44.1.tar.xz
==> Downloading https://gist.githubusercontent.com/jacknagel/af332f42fae80c570a77/raw/7b5fd0d2e6554e9b770729fddacaa2d648327644/glib

curl: (22) The requested URL returned error: 404 Not Found
Error: Failed to download resource "glib--patch"
Download failed: https://gist.githubusercontent.com/jacknagel/af332f42fae80c570a77/raw/7b5fd0d2e6554e9b770729fddacaa2d648327644/glib-hardcoded-paths.diff
=========================================================[ ERROR: Cairo ]==========================================================

LoadError: failed process: Process(`/Users/feldt/.julia/v0.4/Homebrew/deps/usr/bin/brew install --force-bottle staticfloat/juliadeps/glib`, ProcessExited(1)) [1]
while loading /Users/feldt/.julia/v0.4/Cairo/deps/build.jl, in expression starting on line 144

===================================================================================================================================

=========================================================[ BUILD ERRORS ]==========================================================

WARNING: Cairo had build errors.

 - packages with build errors remain installed in /Users/feldt/.julia/v0.4
 - build the package(s) and all dependencies with `Pkg.build("Cairo")`
 - build a single package by running its `deps/build.jl` script

===================================================================================================================================

I'm seeing a different build error:
INFO: Building Homebrew
HEAD is now at 2e6eec2 cmake: update 3.4.0 bottle.
HEAD is now at 5d226de Fix patch locations for glib
INFO: Building Cairo
==> Installing glib from staticfloat/homebrew-juliadeps
==> Downloading https://s3.amazonaws.com/juliacache/glib-2.44.1.tar.xz
Already downloaded: /Users/peter/Library/Caches/Homebrew.jl/glib-2.44.1.tar.xz
==> Downloading https://raw.githubusercontent.com/Homebrew/patches/59e4d32/glib/hardcoded-paths.diff
Already downloaded: /Users/peter/Library/Caches/Homebrew.jl/glib--patch-a4cb96b5861672ec0750cb30ecebe1d417d38052cac12fbb8a77dbf04a886fcb.diff
==> Downloading https://gist.githubusercontent.com/jacknagel/9835034/raw/282d36efc126272f3e73206c9865013f52d67cd8/gio.pa

curl: (22) The requested URL returned error: 404 Not Found
Error: Failed to download resource "glib--patch"
Download failed: https://gist.githubusercontent.com/jacknagel/9835034/raw/282d36efc126272f3e73206c9865013f52d67cd8/gio.patch

this patch file doesn't seem to exist.

cc @ranma42

Yet another Cairo build failure?
julia> Pkg.build("Cairo")
INFO: Building Homebrew
Already up-to-date.
INFO: Building Cairo
Warning: staticfloat/juliatranslated/pixman-0.34.0 already installed
Warning: Already linked: /Users/ortner/.julia/v0.5/Homebrew/deps/usr/Cellar/pixman/0.34.0
Warning: staticfloat/juliatranslated/libpng-1.6.26 already installed
Warning: Already linked: /Users/ortner/.julia/v0.5/Homebrew/deps/usr/Cellar/libpng/1.6.26
Warning: staticfloat/juliatranslated/freetype-2.7 already installed
Warning: Already linked: /Users/ortner/.julia/v0.5/Homebrew/deps/usr/Cellar/freetype/2.7
Warning: staticfloat/juliatranslated/icu4c-58.1 already installed
Warning: Already linked: /Users/ortner/.julia/v0.5/Homebrew/deps/usr/Cellar/icu4c/58.1
Warning: staticfloat/juliatranslated/libffi-3.0.13 already installed
Warning: Already linked: /Users/ortner/.julia/v0.5/Homebrew/deps/usr/Cellar/libffi/3.0.13
Warning: staticfloat/juliatranslated/pcre-8.39 already installed
Warning: Already linked: /Users/ortner/.julia/v0.5/Homebrew/deps/usr/Cellar/pcre/8.39
Warning: staticfloat/juliatranslated/gettext-<IP_ADDRESS> already installed
Warning: Already linked: /Users/ortner/.julia/v0.5/Homebrew/deps/usr/Cellar/gettext/<IP_ADDRESS>
Warning: staticfloat/juliatranslated/glib-2.50.2 already installed
Warning: Already linked: /Users/ortner/.julia/v0.5/Homebrew/deps/usr/Cellar/glib/2.50.2
Warning: staticfloat/juliatranslated/gobject-introspection-1.50.0 already installed
Warning: Already linked: /Users/ortner/.julia/v0.5/Homebrew/deps/usr/Cellar/gobject-introspection/1.50.0
Warning: staticfloat/juliatranslated/pango-1.40.3 already installed
Warning: Already linked: /Users/ortner/.julia/v0.5/Homebrew/deps/usr/Cellar/pango/1.40.3
===================================[ ERROR: Cairo ]===================================

LoadError: Provider Homebrew.HB failed to satisfy dependency pangocairo
while loading /Users/ortner/.julia/v0.5/Cairo/deps/build.jl, in expression starting on line 154

======================================================================================

btw Cairo.jl runs fine on my old 0.3 installation.


Compatibility issue with ./regtest.sh start on macOS

Hello,
I've encountered an issue while running the ./regtest.sh start command from the rgb-lightning-node library:
On a Linux system, the command works successfully without any issues.
However, on macOS, it fails with the following error:
preparing bitcoind wallet
error: timeout on transient error: Could not connect to the server <IP_ADDRESS>:18443
Make sure the bitcoind server is running and that you are connecting to the correct RPC port.
error: timeout on transient error: Could not connect to the server <IP_ADDRESS>:18443
Make sure the bitcoind server is running and that you are connecting to the correct RPC port.
It seems that the script is unable to connect to bitcoind on macOS. Could you please look into adding compatibility for macOS systems? This would greatly enhance the usability of the library across platforms.
Thank you for your work on this amazing project!

At the moment adding compatibility for ARM-based macOS systems is not in our roadmap. The regtest.sh script is meant for testing on x86_64-based Unix systems.
But RLN should run on every system, so if you want to use it on regtest you can just manually run the necessary services locally or in docker.


InvalidClientTokenId: The security token included in the request is invalid

When I ran terraform apply, I got
Error: creating cluster: running eksctl create cluster: /usr/local/bin/eksctl: exit status 1
[ℹ]  eksctl version 0.35.0
[ℹ]  using region us-east-1
Error: checking AWS STS access – cannot get role ARN for current session: InvalidClientTokenId: The security token included in the request is invalid.
status code: 403, request id: 6b4d22f1-0507-45e4-a199-3903df80a0f7
But if I ran eksctl create cluster command directly in the same shell, it worked fine.
What could have caused the issue?

Did this get resolved? I have the same issue


1.5.1 Changelog and Known Issues update

1.5.1 Changelog and Known Issues update

1.5.1 changelog?

Yep, that's right. Title was wrong. Updated!


mysema-commons-lang is unlicenced

The only available source code to mysema-commons-lang is:
http://central.maven.org/maven2/com/mysema/commons/mysema-commons-lang/0.2.4/mysema-commons-lang-0.2.4-sources.jar
This archive only contains files with the following statement:
/*
 * Copyright (c) 2010 Mysema Ltd.
 * All rights reserved.
 */

However, there is no licence for these files; therefore, this dependency is unlicenced and thus illegal to use.
If you’re in a situation to fix that, please obtain and publish an appropriate licence for this artifact; otherwise, it must be removed.
Note that most distributions require such licence statements to be included with the source code.

If you look at the pom
http://central.maven.org/maven2/com/mysema/commons/mysema-commons-lang/0.2.4/mysema-commons-lang-0.2.4.pom
You can see that the project is licensed under The Apache Software License, Version 2.0.
(So I think that they just forgot to update the license headers in the individual files.)
Does that answer your question?
p.s. mysema commons lang is (IIRC) still in the hands of Mysema, but I don't see it in their GitHub organization.
You can try contacting them directly.
p.p.s. @timowest Is it true what I said here? Just want to verify.

The POM file is not necessary either a licence grant or correct. I’ve seen multiple artefacts with misleading or outright wrong information in the POM, so I’m extremely wary of trusting that information.
I’ll try to contact Mysema, Inc. then. Thanks!

(So I think that they just forgot to update the license headers in the individual files.)

I wasn't implying that the POM file is correct, I was merely stating that looking at the POM file the probably intended to license it under ASL 2.0.
Do come back when you know more, and if they want to resolve it.
Then we can depend on the new artifact, if they choose to publish that.

I have a response from an employee of theirs who stated that they indeed intend to have the Apache 2 licence apply to the artefact, but that they cannot issue a new release quickly. I asked them to post here (ideally PGP-signed, to establish some kind of verifyable trust) stating so, but I guess that’s enough for the project the team I’m in is working on at the moment.

For a fast resolve I can verify the Apache 2 license apply for mysema-commons-lang.

Ahh. Then my hunch was correct after all.
Yes, at least some verification of intent would I think be nice.
But maybe it can be put up somewhere a little less volatile than a comment in GitHub Issues?
@mangolas at least thank you for joining the conversation, much appreciated!

It might be worth enabling apache rat to check the license files. I don't know if there is a maven plugin for it though.

I don't know if there is a maven plugin for it though.

It looks like there is.

The best thing is to do it right and put the sources back online, they were in SVN and now exists only in backups, so we would need to republish the project in github. That all can be done, but we just need to find bit time to do it.

Thanks for the official statement; I think the issue can be closed for now.
@mangolas I’d be happy to do the svn to git conversion for you if you can send me the part of the backup

@mirabilos Hah, ok, we first have to locate the backup, which is the real job :)

@mangolas can't you use the src artifact of the last release for this? Seeing your remark is 2 years old and the source still not being available.

Hi I'm facing the same issue as @mirabilos. This package has no repository and no licence available. This is a big issue since it could put you in troubles.
It's a small library with 1 interface and 6 small classes from 2012:

I'm almost sure the utility classes it provides are available either in recent version of the JDK or apache libs. Is there any plan to remove this dep ?
Thanks


fix GEARPUMP-109: add wordcount Java/Scala and DSL/low level API examples

could you help to fix this ?
- _site/gearpump-internals.html
  *  linking to internal hash #what-happen-when-executor-crash that does not exist (line 252)
- _site/get-gearpump-distribution.html
  *  internally linking to /download.html, which does not exist (line 144)
htmlproof 2.6.1 | Error:  HTML-Proofer found 2 failures!

honestly, the title "write your 1st gearpump app" is not as straightforward as "examples". What do you think ?

Hi @manuzhang,

for the "/download.html" link issue, I have fixed in PR #2.
for the rest, I will try to fix it.

For the name "write your 1st Gearpump app", I prefer it over "example".

[Current coverage][cc-pull] is 51.97%

Merging [#6][cc-pull] into [master][cc-base-branch] will increase coverage by +0.60%

@@             master         #6   diff @@
==========================================
  Files           562        562
  Lines          9285       9285
  Methods        8551       8551
  Messages          0          0
  Branches        408        408
==========================================
+ Hits           4768       4825    +57
+ Misses         4517       4460    -57
  Partials          0          0

2 files (not in diff) in ...ts/distributeservice were modified. more

Misses +9
Hits -9

1 files (not in diff) in examples were modified. more

Misses +6
Hits -6

4 files (not in diff) in ...ala/io/gearpump/util were modified. more

Misses -16
Hits +16

2 files (not in diff) in ...rpump/cluster/client were modified. more

Misses -57
Hits +57

6 files (not in diff) in .../io/gearpump/cluster were modified. more

Misses -19
Hits +19

1 files (not in diff) in ...in/scala/io/gearpump were modified. more

Misses +13
Hits -13

File .../worker/Worker.scala (not in diff) was modified. more

Misses +7
Partials 0
Hits -7

Powered by Codecov. Last updated by caec5b9
[cc-base-branch]: https://codecov.io/gh/apache/incubator-gearpump/branch/master?src=pr
[cc-pull]: https://codecov.io/gh/apache/incubator-gearpump/pull/6?src=pr

I don't think the failure is related to this PR as my PR is only document related.

LGTM + 1

please rebase

rebase done.


[SPARK-33449][SQL] Add File Metadata cache support for Parquet and Orc

What changes were proposed in this pull request?
The main purpose of this pr is to introduce the File Meta Cache mechanism for Spark SQL and the basic File Meta Cache implementation for Parquet and Orc is provided at the same time.
The main change of this pr as follows:

Defined a FileMetaCacheManager to cache the mapping FileMetaKey to FileMeta. The FileMetaKey is the cache key, equals is determined by the file path by default. The FileMeta used to represent the cache value and It is generated by the  FileMetaKey#getFileMeta   method.

Currently, the FileMetaCacheManager supports a simple cache expiration elimination mechanism, and the expiration time is determined by the new config  FILE_META_CACHE_TTL_SINCE_LAST_ACCESS

For Parquet file format, this pr added ParquetFileMetaKey  and ParquetFileMeta to cache Parquet file Footer and the Footer cache can be used by Vectorized read scene both in DS API V1 and V2, the feature will be enabled when FILE_META_CACHE_PARQUET_ENABLED  is true

For Orc file format, this pr added OrcFileMetaKey  and OrcFileMeta to cache Orc file Tail and and the Tail cache can be used by Vectorized read scene both in DS API V1 and V2, the feature will be enabled when FILE_META_CACHE_ORC_ENABLED  is true

Currently, the file meta cache mechanism cannot be used by RowBasedReader, and it needs the completion of PARQUET-1965 and ORC-746 for further support.
Why are the changes needed?
Support Parquet and Orc datasource use File Meta Cache mechanism to reduce the times of metadata reads multiple queries are performed on the same dataset.
Does this PR introduce any user-facing change?
Add 3 new config:

FILE_META_CACHE_PARQUET_ENABLED(spark.sql.fileMetaCache.parquet.enabled) to indicate if enable parquet file meta cache mechanism
FILE_META_CACHE_ORC_ENABLED(spark.sql.fileMetaCache.orc.enabled) to indicate if enable orc file meta cache mechanism
FILE_META_CACHE_TTL_SINCE_LAST_ACCESS(spark.sql.fileMetaCache.ttlSinceLastAccess) to represent Time-to-live for file metadata cache entry after last access, the unit is seconds.

How was this patch tested?

Pass the Jenkins or GitHub Action
Add new test suites to OrcQuerySuite and ParquetQuerySuite

@wangyum  WIP now, missing some configuration entries, test suite and orc file support

Test build #131655 has started for PR 30483 at commit 8357771.

Test build #131655 has finished for PR 30483 at commit 8357771.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131655/

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36256/

Since your fix is merged here,
could you fix Scala style and rebase to the master, @LuciferYang ?

OK~ will do it later ~

Test build #131754 has started for PR 30483 at commit 8bba51a.

Test build #131754 has finished for PR 30483 at commit 8bba51a.

This patch fails Java style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class TruncateTable(

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131754/

@LuciferYang It would be great if we had some benchmark numbers.

Test build #131760 has started for PR 30483 at commit 3e2db1a.

@wangyum this is a very good suggestion ~

Test build #131771 has started for PR 30483 at commit 92d2f37.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36352/

Test build #131775 has started for PR 30483 at commit 44ca052.

Test build #131760 has finished for PR 30483 at commit 3e2db1a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131760/

Test build #131779 has started for PR 30483 at commit 7254d88.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36358/

Test build #131782 has started for PR 30483 at commit bc25c4e.

Test build #131771 has finished for PR 30483 at commit 92d2f37.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131771/

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36373/

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36369/

Test build #131775 has finished for PR 30483 at commit 44ca052.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Build finished.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131775/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36377/

Test build #131782 has finished for PR 30483 at commit bc25c4e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class OrcFileMeta(tail: OrcTail) extends FileMeta
class ForTailCacheReader(path: Path, options: OrcFile.ReaderOptions)

Test build #131779 has finished for PR 30483 at commit 7254d88.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Build finished.

Build finished.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131779/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131782/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36380/

@LuciferYang . In general, I have one concern. Is this robust on Parquet/ORC version changes? I'm wondering if this will block Apache Spark from upgrading them.
Apache Parquet 1.12 has some incompatible changes and Apache Spark is still on Parquet 1.10.
Apache ORC 1.6.0 has also incompatible API changes and Apache Spark is still using ORC 1.5.

More research work may be needed to determine if it is appropriate to continue this pr at this time, do you have any suggestions on this @wangyum ?

In addition, in order for RowBasedReader to use file meta cache, we certainly need to change the API of Apache Parquet and Apache ORC and upgrade the version.

Could you file the corresponding JIRA to Apache Parquet and Apache ORC community and link them in the JIRA and the PR description?

we certainly need to change the API of Apache Parquet and Apache ORC and upgrade the version.

Test build #133176 has started for PR 30483 at commit 190dc8a.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37775/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37775/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37775/

Test build #133176 has finished for PR 30483 at commit 190dc8a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133176/

Test build #133731 has started for PR 30483 at commit c485cc5.

Test build #133731 has started for PR 30483 at commit c485cc5.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38319/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38319/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38319/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38319/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38319/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38319/

Test build #133731 has finished for PR 30483 at commit c485cc5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #133731 has finished for PR 30483 at commit c485cc5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133731/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133731/

Test build #134351 has started for PR 30483 at commit c485cc5.

Test build #134351 has started for PR 30483 at commit c485cc5.

Test build #134351 has finished for PR 30483 at commit c485cc5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134351 has finished for PR 30483 at commit c485cc5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134351/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134351/

Thank you very much for your review @dongjoon-hyun , there are other things to do this week, I'll update this PR later

Thank you very much for your review @dongjoon-hyun , there are other things to do this week, I'll update this PR later

No problem, @LuciferYang . :)

No problem, @LuciferYang . :)

Test build #134438 has started for PR 30483 at commit b872010.

Test build #134438 has started for PR 30483 at commit b872010.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39023/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39023/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39023/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39023/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39023/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39023/

Test build #134439 has started for PR 30483 at commit 0a224ff.

Test build #134439 has started for PR 30483 at commit 0a224ff.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39025/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39025/

Test build #134444 has started for PR 30483 at commit 98ef2de.

Test build #134444 has started for PR 30483 at commit 98ef2de.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39030/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39030/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39025/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39025/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39025/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39025/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39030/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39030/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134444/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134444/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39030/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134439/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134439/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39030/

Test build #134438 has finished for PR 30483 at commit b872010.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134438 has finished for PR 30483 at commit b872010.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134438/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134438/

Could you file the corresponding JIRA to Apache Parquet and Apache ORC community and link them in the JIRA and the PR description?

PARQUET-1965 and ORC-746

Could you file the corresponding JIRA to Apache Parquet and Apache ORC community and link them in the JIRA and the PR description?

PARQUET-1965 and ORC-746

Simple test:
val df = spark.read.parquet(or orc)("file:/home/work/xxx/data")

df.createOrReplaceTempView("test_table")

spark.sql("select sum(a), sum(b), sum(c) from test_table where id = 1381339").show
spark.sql("select sum(a), sum(b), sum(c) from test_table where id = 28643411").show

Data Source V1:

parquet with spark.sql.fileMetaCache.parquet.enabled =false

parquet with spark.sql.fileMetaCache.parquet.enabled =true

orc with spark.sql.fileMetaCache.orc.enabled =false

orc with spark.sql.fileMetaCache.orc.enabled =true

Test build #134816 has started for PR 30483 at commit 120678d.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39404/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39404/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39404/

Test build #134816 has finished for PR 30483 at commit 120678d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134816/

@LuciferYang, where footer is cached, driver or executor?  As I understand, the footer will be used at executor side, are you caching the footer at executor side?

@baibaichen footer will cached in Executor side,  At present, the original scheduling strategy is reused, and the sensitive scheduling of footer cache is not considered

Test build #135493 has started for PR 30483 at commit eb8fa71.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40074/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40074/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40074/

Test build #135493 has finished for PR 30483 at commit eb8fa71.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135493/

I removed the Stale tag and reopen this for Apache Spark 3.3.

Test build #142183 has started for PR 30483 at commit eb8fa71.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46695/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46695/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46695/

Hi, @LuciferYang . Are you still interested in this?

Test build #142183 has finished for PR 30483 at commit eb8fa71.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142183/

Hi, @LuciferYang . Are you still interested in this?

Yes, I'm still interested in it. I'll try to update it to master first. However, since ParquetFileReader no longer has a non deprecated constructor support pass footer, we need to make a decision together.

Test build #142214 has started for PR 30483 at commit 61175ed.

Test build #142215 has started for PR 30483 at commit 179e7b0.

Test build #142219 has started for PR 30483 at commit 36f502e.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46727/

Test build #142219 has finished for PR 30483 at commit 36f502e.

This patch fails Java style tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142219/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46727/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46727/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46731/

Test build #142223 has started for PR 30483 at commit 850c52b.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142215/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46735/

Test build #142214 has finished for PR 30483 at commit 61175ed.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142214/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142223/

Test build #142273 has started for PR 30483 at commit 630f8db.

I have some problems now:

How should we pass the filemeta to the Reader? Now I add a new API named setCachedXXX()
Parquet no longer has non Deprecated API that can be used to pass footer to create new ParquetFileReader
Should we add more configs related to eviction policy, such as maximumSize
Should scheduling sensitivity be considered at present？

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46781/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46781/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46781/

Test build #142273 has finished for PR 30483 at commit 630f8db.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142273/

In other words, can you spin off ORC-only PR?

OK, I will create a new Jira  and  give a ORC-only pr ~

Test build #142494 has started for PR 30483 at commit aace310.

Test build #142494 has finished for PR 30483 at commit aace310.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142494/

@dongjoon-hyun Because https://github.com/apache/spark/pull/33748 gives an ORC-only pr, I'll change this PR to Parquet-only

There will be some duplicate codes in the two PR, and this part of the code will be synchronized after one of them is merged

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46992/

Test build #142498 has started for PR 30483 at commit fa75a95.

Test build #142498 has finished for PR 30483 at commit fa75a95.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142498/

Got it. Thank you for moving forward this efforts, @LuciferYang .

Test build #142500 has started for PR 30483 at commit 4c022d7.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46992/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46992/

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47001/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46999/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47001/

Test build #142505 has started for PR 30483 at commit 104b125.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46999/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47006/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46999/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47006/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47006/

Test build #142500 has finished for PR 30483 at commit 4c022d7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142500/

Test build #142505 has finished for PR 30483 at commit 104b125.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class FileMetaKey
case class ParquetFileMetaKey(path: Path, configuration: Configuration)
class ParquetFileMeta(val footer: ParquetMetadata) extends FileMeta

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142505/


Expose all error constructors

This PR tackles issue #310.
I propose to make all error types public, similar to how domain_error and others were previously made accessible. Given our project's significant reliance on ichiban/prolog, having access to the entire spectrum of error types used by the library would be highly beneficial for us as developers.
This means making these error constructors available for everyone: existence_error, permission_error, representation_error, resource_error, syntax_error, and evaluation_error.
Thanks.

@ichiban Thanks for the code review and your insightful comments. Much appreciated. I've made the updates.


Hand Menu example eye tracking does not function well over Holographic Remoting

Describe the bug
When the user is in the Hand Menu example scene over Holographic Remoting and they raise their flat palm into view of the device sensors without looking at their palm, the hand menu will populate on the user's flat palm. This issue only happens with XR SDK and Holographic Remoting.  I do not know if this issue effects OpenXR as well since articulated hands do not work over Holographic Remoting.
To reproduce
Steps to reproduce the behavior:

Create a new project, import the MRTK, and enable the XR SDK
Open the Hand Menu Examples scene
Change to the 'DefaultHoloLens2XRSDKConfigurationProfile'
Start the Holographic Remoting session
Raise a flat palm into view of the device sensors but do not look in the direction of your flat palm
See the hand menu populate on your flat palm, even though your gaze was not near your flat palm

Your setup (please complete the following information)

Unity Version 2020.3.2f1
MRTK Version 2.5.1

Target platform (please complete the following information)

HoloLens 2
UWP
XR SDK

By default eye tracking is turned off in the XR SDK profile while we wait on some Unity bugs to be fixed.

We appreciate your feedback and thank you for reporting this issue.
Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.
Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.
Please note that MRTK3 was released in August 2023. It features an all new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visithttps://www.mixedrealitytoolkit.org.
Thank you for your continued support of the Mixed Reality Toolkit!


Adapt default config for flashcam, bring in line with current grid production

After #2188, we can adjust the default cleaning configuration for flashcam back to the same as for NectarCam
Also adjusted some other settings to the same values we used for the latest grid production.

As this "enables" (sort of) FlashCam it is probably worth having a changelog entry for it?

There is one aleady for this from the flashcam PR. I think that's sufficient. I will have a look when rendering the full changelog for 0.19


webpack-dev-server can't use

Bug report
use webpack 5.1.3 and webpack-dev-server found a mistack:
cmd console:
Error: Cannot find module 'webpack-cli/bin/config-yargs'
Require stack:

F:\webpack-demo\node_modules\webpack-dev-server\bin\webpack-dev-server.js
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:831:15)
at Function.Module._load (internal/modules/cjs/loader.js:687:27)
at Module.require (internal/modules/cjs/loader.js:903:19)
at require (internal/modules/cjs/helpers.js:74:18)
at Object. (F:\webpack-demo\node_modules\webpack-dev-server\bin\webpack-dev-server.js:65:1)
at Module._compile (internal/modules/cjs/loader.js:1015:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1035:10)
at Module.load (internal/modules/cjs/loader.js:879:32)
at Function.Module._load (internal/modules/cjs/loader.js:724:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12) {
code: 'MODULE_NOT_FOUND',
requireStack: [
'F:\webpack-demo\node_modules\webpack-dev-server\bin\webpack-dev-server.js'
]
}

Other relevant information:
webpack version:  5.1.3
Node.js version: 12.19.0
Operating System: win10
Additional tools:webpack,webpack-cli,Chrome

Please open webpack-dev-server repo and look at pinned issue

https://github.com/webpack/webpack-dev-server/issues/2424


[SPARK-30762] Add dtype=float32 support to vector_to_array UDF

What changes were proposed in this pull request?
In this PR, we add a parameter in the python function vector_to_array(col) that allows converting to a column of arrays of Float (32bits) in scala, which would be mapped to a numpy array of dtype=float32.
Why are the changes needed?
In the downstream ML training, using float32 instead of float64 (default) would allow a larger batch size, i.e., allow more data to fit in the memory.
Does this PR introduce any user-facing change?
Yes.
Old: vector_to_array() only take one param
df.select(vector_to_array("colA"), ...)

New: vector_to_array() can take an additional optional param: dtype = "float32" (or "float64")
df.select(vector_to_array("colA", "float32"), ...)

How was this patch tested?
Unit test in scala.
doctest in python.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

ok to test

Test build #118155 has started for PR 27522 at commit 6e79d90.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22917/
Test PASSed.

Test build #118158 has started for PR 27522 at commit 398d483.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22920/
Test PASSed.

Test build #118155 has finished for PR 27522 at commit 6e79d90.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118155/
Test FAILed.

Test build #118159 has started for PR 27522 at commit 9df802a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22921/
Test PASSed.

Test build #118158 has finished for PR 27522 at commit 398d483.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118158/
Test FAILed.

Test build #118159 has finished for PR 27522 at commit 9df802a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118159/
Test FAILed.

Test build #118165 has started for PR 27522 at commit 603935d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22926/
Test PASSed.

Test build #118165 has finished for PR 27522 at commit 603935d.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118165/
Test FAILed.

Test build #118167 has started for PR 27522 at commit 32b4b4d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22928/
Test PASSed.

Test build #118169 has started for PR 27522 at commit 60e4b99.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22930/
Test PASSed.

Test build #118167 has finished for PR 27522 at commit 32b4b4d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118167/
Test FAILed.

Test build #118169 has finished for PR 27522 at commit 60e4b99.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118169/
Test FAILed.

Test build #118190 has started for PR 27522 at commit 6363bdc.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22952/
Test PASSed.

Test build #118190 has finished for PR 27522 at commit 6363bdc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118190/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22993/
Test PASSed.

Test build #118233 has started for PR 27522 at commit 29fe58b.

Test build #118233 has finished for PR 27522 at commit 29fe58b.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118233/
Test FAILed.

Test build #118241 has started for PR 27522 at commit e165cd7.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23001/
Test PASSed.

Test build #118241 has finished for PR 27522 at commit e165cd7.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118241/
Test FAILed.

Test build #118244 has started for PR 27522 at commit 9fcdd1f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23004/
Test PASSed.

ok to test

Test build #118244 has finished for PR 27522 at commit 9fcdd1f.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118244/
Test FAILed.

Test build #118249 has started for PR 27522 at commit dc3da25.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23008/
Test PASSed.

Test build #118249 has finished for PR 27522 at commit dc3da25.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118249/
Test PASSed.

Also update the PR description.

Test build #118303 has started for PR 27522 at commit 9667a04.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23061/
Test PASSed.

Test build #118303 has finished for PR 27522 at commit 9667a04.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118303/
Test PASSed.

Test build #118338 has started for PR 27522 at commit dbcbcfa.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23095/
Test PASSed.

Test build #118339 has started for PR 27522 at commit 3ae3368.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23096/
Test PASSed.

Test build #118338 has finished for PR 27522 at commit dbcbcfa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118338/
Test PASSed.

Test build #118339 has finished for PR 27522 at commit 3ae3368.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118339/
Test PASSed.

Test build #118348 has started for PR 27522 at commit 848220d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23105/
Test PASSed.

Test build #118348 has finished for PR 27522 at commit 848220d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118348/
Test PASSed.

Merge to master/branch-3.0


Update jQuery Library to latest version

What kind of issue is this? (put 'x' between the square brackets)

[ ] Question. This issue tracker is not the place for questions. If you want to ask how to do
something, or to understand why something isn't working the way you expect it to, use
http://stackoverflow.com/questions/ask .
Provide working code, starting from http://jsfiddle.net/JNsu6/15/.
We monitor the tag featherlight.js.

[ ] Bug report. If you’ve found a bug, you must provide a minimal example in a CodePen,
starting from http://jsfiddle.net/JNsu6/15/ .

[X] Feature Request. Make sure there's no good way to do what you want first;
consider asking on http://stackoverflow.com/questions/ask first.

I'm sorry, I'm not sure I understand your request. You should be free to use mostly any version of jQuery you'd like.


Why push notifications won't work.

Why you're hearing about it now
When I saw the initial spec sheet for this app I was under the impression this was a single deployment SAAS product, but after seeing some mockups it quickly became apparent this was not the case.  I assumed someone had come up with a solution for this before it was put on the spec sheet.  Again, this is apparently not the case, so I figured now would be a good time to explain it.
How APNS Works (Briefly)
APNS (Or Apple Push Notification Service) is an inflexible beast.  But the root of the problem comes from the fact that Project Jellyfish is meant to be deployed individually by the Corporations choosing to use it.  We'll leave the issues of connecting to APNS from behind a corporate firewall/DMZ aside for now, but the connection requires a one to one paired SSL certificate created in the Apple Developer Program account of the App's (iOS) publisher (which in this case is BAH).
– You create these SSL CSRs, upload them, and Apple returns you a valid SSL certificate that validates against their own Root CAs for which the creator is the sole owner of the Public/Private key pair that backs them.
– Apple does not have a copy, and if you lose the key you have to redeploy the app.
– This SSL certificate is required for the Server backend to connect to APNS (which is a socket connection) and cannot be avoided.
– In the App (iOS), the user must consent to receive push notifications, at which point a request will go to Apple to retrieve a semi-unique (Unique Per App and Device) identifier. Normally, this string is sent back to the backend service for storage and pairing with the User's account in the backend.
– When the backend chooses to send a notification to a user's device, this identifier is used in the body of data sent to APNS.
– When that body of data arrives at APNS, Apple then sends it along to the device. This is one of the highest performance web services in the world and the lag between the original backend sending the request and the push notification arriving on device is measured in milliseconds, even on cellular connections.
The Problem
Since APNS requires a SSL certificate to receive a connection from the backend, having individual backends sending requests directly to APNS is not possible, as you would have to share this SSL certificate.  This is not only against Apple's ToS but also the same thing as not using SSL in the first place.  The SSL certificate ensures that if someone's cloud deployed instance of Project Jellyfish is compromised, it cannot be used to send spam notifications without the password that decrypts the private key backing the SSL certificate.  However, since the backend is written in Ruby and not compiled, that password will be in plain text somewhere, but we'll gloss over that for the time being.
Is there a solution?
A possible solution for this would be for BAH to provide, maintain, and ensure uptime of a closed source relay to APNS.  An Operator of Project Jellyfish would register with you, and when the App (iOS) goes to send the aforementioned unique identifier back to the Operator, the Operator will instead send a heavily authenticated POST request to your relay which will forward the request to APNS when a notification is to be sent.
More Information
– APNS
– Amazon AWS SNS

This isn't an issue per se.  We'll need to use Amazon SNS.  Going to close this ticket.


Need a better unit  to set CPU for deis limits

Cpu limits were designed keeping docker cli in mind which takes cpu shares from 0-1024.
For K8s we are converting these CPU shares to number of CPU by dividing it with 1024.
and we only allow integer numbers.
Specifying CPU shares and converting them to #CPUS is not a good model in a heterogeneous cluster, where machines might have different #cpus. Also change deis limits from integer to float for assinging more granular limits to apps.
https://github.com/kubernetes/kubernetes/blob/master/docs/design/resources.md#resource-types
https://github.com/kubernetes/kubernetes/blob/master/docs/design/resources.md

yes, I agree that we should just go with specifying the number of CPUs available in float so the data sent in is a 1:1 relationship with how kubernetes handles limit request. We did it the old way so people using v1 wasn't confused when switching from fleet to kubernetes, but now that we're all-in it makes sense to follow their model.

Discussing this offline, we thought that this might be the friendliest implementation:
$ deis limits:set -c cmd=50%  # allocates half of a node's CPUs, as 512 shares did before
$ deis limits:set -c cmd=2    # allocates exactly two of a node's CPUS

My bad there is no way to set CPU shares any more in kubernetes. We can only set #CPUS or milli CPU
units. the client commands are like.
$ deis limits:set -c cmd=500m  # allocates 512 shares  or 500 milli units of 1 core cpu.
$ deis limits:set -c cmd=2    # allocates exactly two of a node's CPUS


Horizontal Scroll + Ember-Scrollable

Opening this issue to track the progress of alphasights/ember-scrollable#29
cc @alexander-alvarez.

Are there any updates on this since #321 has been merged?
I would really like to see horizontal scrolling work with a fixed header.

@KeithClinard this is already supported (see #222). This issue is about using ember-scrollable's virtual scrollbars instead of the current rudimentary solution.

As it appears to be unrelated to this issue, I've created another issue for the problem I am having. (#335)

I see support for both horizontal and vertical scrolling was recently merged into ember-scollable. What's the timeline on integrating it with ember-light-table?

Bump
Is horizontal scroll supported?
I tried adding ember-scrollable to wrap the light-table, but it didn't work.
{{#ember-scrollable}}
  Some long content...
{{/ember-scrollable}}

Any help is appreciated. Thanks

It's not currently. There's a quite a bit of work to be done with freezing rows/columns/etc which makes the horizontal scroll complete. That I know of, it isn't being tackled at the moment


Multiple contraptions causes huge lag spikes 1.20.1 FABRIC

Describe the Bug
It's on Minecraft 1.20.1 FABRIC
I have around 10 train contraptions circling around an area around 32+ chunks.
It's on a hosted server.
The game run at an average of 80 fps. But the fps drops down to 0/20 for 20/30 secs then returns to normal.
This happens everywhere in this 32 chunks area, around all 5/10 minutes (it's reccurent but random).
It's the same lag I have when I log on the server  and the game is loading the chunks.
I tried to optimize it with better chunk loading, which works fine, like the log lag have almost disapeard, but this is issue is still there.
I also tried using entity distance. I tried with the minimal and the maximal render distance (up to 512 blocks). The bug was still there in both configurations. So it's not a render issue I'd say.. and the fps were still good overhaul.
As it is now, it looks to me like a chunk loading/unloading issue with the contraption (this bug doesn't happens in area without contraptions).
I have put together all the optimisation mods I could find but yet nothing have yet seems to do a real difference for this issue.
I tried to optimise the cpu render thread, did nothing, the same for the ram, I went from 28gb to 8gb hoping it was a kind of memory issue (too uch ram isn't good) but it did nothing.
Also the server is on OMG-Server, I can't modify the ram nor the Java argument. The server is supposed to dynamically gives enough ram to the game.
I have many mods, but everything else than this area is running just fine.

Reproduction Steps

have many trains contraptions circling in a 32 chunks area
wait for some minutes
experience huge lags for up to 30 secs
...

Expected Result
The game not lagging randomly
Screenshots and Videos
No response
Crash Report or Log
No response
Operating System
Windows 11
Mod Version
0.5.1f
Minecraft Version
1.20.1
Other Mods
No response
Additional Context
No response

shaders disable flywheel it's likely caused by that, try without shaders

It still the same without shaders, and I have the flywheel compatibility

performance reports are not helpful without running a profiler or providing exact reproduction steps.


Fix not inferring types through := operator in release builds

Fixes https://github.com/godotengine/godot/issues/42124

@akien-mga It's nice, that it was finally fixed, but I feel bad that my PR was ignored for months and suddenly @vnen pushed his own, exact same change to repository without closing the issue I mentioned above https://github.com/godotengine/godot/commit/a3fe028809f805b19c41683087427e7d88440519

As we discussed on Rocket.Chat, this was sadly overlooked because the referenced issue #42124 was not linked to its duplicate reports (#48090, #48999). I asked @vnen to fix those as they seemed critical and we missed that a PR pending review was already doing the same. Sorry about that, and thanks for the contribution nevertheless!


As a user, I want to filter library items by borrowing activity dates so that I can see items that circulated in the library in a particular time period.

@jkotin similar question to the one I just asked on #231 . This story references "borrowing activity" and in the design the filter is labeled "Borrowing Dates". Should I only include dates from borrowing events, or should it include all book related events? And if the latter, do we need to change the label?

All book related events is my view.

YES, when I filter by circulation dates (like 1919-1920), I get a list of books that circulated in that time period. If I click through to a certain book, I see that the top results will be for those circulation dates, and then I see when else the book circulated (who borrowed it and when). I also tried 1930-1940.


Can almost not read messages

Subject: Can almost not read messages
Description:
The layout of the website
Results:
I got a message in another language and I wanted to select it so I could translate it. But the layout is so that name of the other person and the place where I can type the response are almost filling the screen and only 1 row is left to read the messages.
This is on Android webbrowser (opera)
Make the name a bit smaller.
I have a screenshot
Backend version: develop-71038cd3
Frontend version: develop-71038cd3
User Agent: Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Mobile Safari/537.36 OPR/<IP_ADDRESS>
Page: https://couchers.org/messages/request/98280
User: aladin (4384)

Thanks @Cuddle-sheep ! Was just going to ask for a screenshot. Yeah we need to redo the way this screen is laid out.

@aapeliv current behavior disables messaging within the request thread to prevent annoying/retaliatory messages in case of a declined request.
We should just allow messages to be sent within the request thread even after a request is declined. However, before we enable this we should have a "block user" button in place in case people get those annoying/retaliatory messages. The block user feature will also address other issues and is a common feature request. Block user issue is already created: #4165

@jesseallhands This is related to the layout, not the responding to closed requests question.

@aapeliv your welcom.
redo of the layout would be great!
@jesseallhands I also agree on your point. being abel to sent a thank you or somthing after somone diclince but aapeliv is right that this isue is about the layout.

Now tracked in #5055

Sorry, #5028


mermaidjs diagram arrows are too dark in dark mode

If you do doctave init and serve the default example.md, the sample diagram's arrows are too dark in dark mode, making it hard to see.

I also noticed that reloading the page (after enabling dark mode) makes it the reverse --- the arrows are bright but the nodes are too dark, making the nodes hard to read.

Thanks for reporting this. Maybe we can tweak the default theme a bit to make these more readable.
We're also going to bump the Mermaid JS version (#17). We may want to do this at the same time.