[6.x] Append database name for withCount

If the database name for the relationship isn't the same as the current, append it to the table name for the count subquery. Addresses #23042

Will update the test mock objects to reflect the changes.

@ollieread feel free to reattempt when tests are passing.

@driesvints I've found an inconsistency here. The mock object is created from ConnectionInterface which doesn't have a getDatabaseName() method. However, this method is called in at least 5 different places.
It is used:

When getting a doctrine connection
When getting a MySQL column listing
When check if MySQL has a table
When dropping all SQLite tables
When refreshing SQLite database files

Since all connections extend Illuminate\Database\Connection they all have the method available. There are third party implementations of the interface that do not add this method, but there are also some that do. I think the ideal way to address this would be to add the method to the interface, though that's going to be a breaking change.

@ollieread any breaking change will need to go to master.

@ollieread thanks for attempting this btw!

No problem at all @driesvints. Should I put these as two separate PRs, with the introduction of the getDatabaseName method as a separate thing?

Yeah I think the interface change can be a separate PR apart from this one 👍


Master

Added calendar-mode to toggle month picker as well as day picker.

Let me know if there is anything I can do to help make this pull request easier (if possible).

Nice, I'm just wondering if we should create a real fork and use it only as MultipleMonthPicker because there are several options available for date picking that we can't use for month picking right now.
I think it would be better to have 2 directives, one for dates as mine and one for month.
If you agree you can create a fork of MultipleDatePicker, change everything to fit month picking only and I'll add a link in the readme to your fork.

So you're thinking something like MultipleMonthPicker and MultipleDatePicker?

Yes

Alright - I'm closing.

Updated and put on https://github.com/Xal3ph/MultipleMonthPicker  Also published to bower


Conditional Add To BuiltinDataTypeManager

It would be nice if it we're possible to have classes which end in DataType to be added to the built-in datatype manager only under specific conditions. This can be achieved in a similar fashion to the analyzers canAnalyze function.
The most immediate use case of this is for the RTTI data types. Currently the RTTI builtin datatypes are windows specific and do not fit the ones used by the GNU compilers and the Itanium CXX ABI. I have been working on the DataTypes for the Itanium abi rtti the past few weeks and it would be nice to be able to have them appear only when the binary is elf format. The same would follow for PE binaries and the RTTI DataTypes.
It appears that they are loaded upon startup. Maybe instead they could just be hidden under certain conditions.

I don't think this is necessary anymore.  I see a filter exists and is likely sufficient.


Feature/examples

Add examples of dataset generation and modification using transforms, trainings with TensorFlow and PyTorch

@imshashank could you review this?

This looks great, can we merge??


Changed let to var

Changed let to var for config

Please review it. I've never worked with swift before and I have no way to test locally.


Create interface for generating strategy

Create interface for generating strategy with appropriate exception(s) and factory.

Introduced dependencies holder which will be finished in #42.


add redis ticket registry

module:
cas-server-support-redis-ticket-registry
Please check, thank you!!!

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Very nice! Thank you.

Ok, I will try to fixed .

Lots better. Thanks.
Next, you want to write some docs. Add a new MD page similar to other registries. add a link to it in the sidebar, and document what the registry is, what it does, why it does it, etc. Follow the same pattern as other docs.

Dont forget to also update the properties in the docs as well. List them all under a relevant title.

This looks perfectly ready to me. Just remember to update docs and post back. Should be good to merge.

Hello. Have you had a chance to add docs for this PR?

Which one I should be added to the branch ?I didn't see the doc about cas5 branch , Later I will add 👍

if you look at the cas-server-documentation module, you will find markdown files that are the docs for the master branch (which is cas 5). Use those as a sample. You may want to start by copying a doc that discusses a ticket registry into a new file for redis and add content. When you're done, make sure you also update the sidebar to include a reference to the new page.
Also, given our release policy this change will likely go into 5.1. You have about a month or so and I may have to ask you to merge with master once more when we are ready to baseline this.

Ok, I see !

i have add some docs , please review!

hey @izerui could you please merge with master once and resolve conflicts?

ok

I closed it, and in #2233 opened again
thanks!


 It's outdated, unnecessarily complex, doesn't consider Linux users and doesn't cover firmware updates.

Are you planning any updates? Want to add Ukrainian translation

Are you planning any updates? Want to add Ukrainian translation

Not really. While the current wiki has all of those issues, the techniques described in it still work (although keep in mind there's probably a better way of doing a lot of things). Feel free to message me on Discord (bandithedoge#7040) if you want some assistance.


specify <IP_ADDRESS> for hotServer

Specifying <IP_ADDRESS> with the port for the hotServer fixes an issue where a Windows browser was unable to connect to the hotServer running on WSL.

Full URL http://<IP_ADDRESS>:35729/?page=%2Fbud%2Fview%2Findex.svelte

Excellent, thank you @theEyeD!


[Public Catalog] Align QS CTA buttons on details page with In product expeirence.

Relates to https://github.com/newrelic/newrelic-observability-packs/issues/211
We've moved some buttons around in product and want to align the external catalog with the internal catalog.
Acceptance Criteria

[x] Obtain design from @danielgolden
[x] Update button placement to match the mock up
[x] Remove padding from top of right rail

Mock Up 🎨
NOTE: do not adjust the top bar to touch the global header

looks great 👍


How to run this project?

Can you please tell the execution steps required? And how to link OpenCV

Install OpenCV and dlib. Look up the instructions online.

I have opencv and dlib installed. I tried compiling by cding into the directory (after copying the two missing files), and running: g++ *.cpp
g++ *.cpp
In file included from FaceDetectorAndTracker.cpp:1:0:
FaceDetectorAndTracker.h:49:5: error: ‘unique_ptr’ in namespace ‘std’ does not name a type
std::unique_ptrcv::VideoCapture m_camera;
^
FaceDetectorAndTracker.h:54:5: error: ‘unique_ptr’ in namespace ‘std’ does not name a type
std::unique_ptrcv::CascadeClassifier m_faceCascade;
^
FaceDetectorAndTracker.h:90:58: warning: non-static data member initializers only available with -std=c++11 or -std=gnu++11 [enabled by default]
bool                                    m_tracking = false;
^
FaceDetectorAndTracker.h:92:25: warning: non-static data member initializers only available with -std=c++11 or -std=gnu++11 [enabled by default]
size_t m_numFaces = 0;
^
FaceDetectorAndTracker.h:94:63: warning: non-static data member initializers only available with -std=c++11 or -std=gnu++11 [enabled by default]
const double                            m_tmMaxDuration = 2.0;
^
FaceDetectorAndTracker.cpp: In constructor ‘FaceDetectorAndTracker::FaceDetectorAndTracker(std::string, int, size_t)’:
FaceDetectorAndTracker.cpp:13:5: error: ‘m_camera’ was not declared in this scope
m_camera = std::make_uniquecv::VideoCapture(cameraIndex);
^
FaceDetectorAndTracker.cpp:13:16: error: ‘make_unique’ is not a member of ‘std’
m_camera = std::make_uniquecv::VideoCapture(cameraIndex);
^
FaceDetectorAndTracker.cpp:13:49: error: expected primary-expression before ‘>’ token
m_camera = std::make_uniquecv::VideoCapture(cameraIndex);
^
FaceDetectorAndTracker.cpp:20:5: error: ‘m_faceCascade’ was not declared in this scope
m_faceCascade = std::make_uniquecv::CascadeClassifier(cascadeFilePath);
^
FaceDetectorAndTracker.cpp:20:21: error: ‘make_unique’ is not a member of ‘std’
m_faceCascade = std::make_uniquecv::CascadeClassifier(cascadeFilePath);
^
FaceDetectorAndTracker.cpp:20:59: error: expected primary-expression before ‘>’ token
m_faceCascade = std::make_uniquecv::CascadeClassifier(cascadeFilePath);
^
FaceDetectorAndTracker.cpp: In member function ‘void FaceDetectorAndTracker::operator>>(cv::Mat&)’:
FaceDetectorAndTracker.cpp:52:9: error: ‘m_camera’ was not declared in this scope
if (m_camera->isOpened() == false)
^
FaceDetectorAndTracker.cpp:57:6: error: ‘m_camera’ was not declared in this scope
m_camera >> frame;
^
FaceDetectorAndTracker.cpp: In member function ‘std::vector<cv::Rect_ > FaceDetectorAndTracker::faces()’:
FaceDetectorAndTracker.cpp:75:22: error: ISO C++ forbids declaration of ‘face’ with no type [-fpermissive]
for (const auto& face : m_facesRects)
^
FaceDetectorAndTracker.cpp:75:29: error: range-based ‘for’ loops are not allowed in C++98 mode
for (const auto& face : m_facesRects)
^
FaceDetectorAndTracker.cpp:77:39: error: request for member ‘x’ in ‘face’, which is of non-class type ‘const int’
faces.push_back(cv::Rect(face.x * m_ratio.x, face.y * m_ratio.y, face.width * m_ratio.x, face.height * m_ratio.y));
^
FaceDetectorAndTracker.cpp:77:59: error: request for member ‘y’ in ‘face’, which is of non-class type ‘const int’
faces.push_back(cv::Rect(face.x * m_ratio.x, face.y * m_ratio.y, face.width * m_ratio.x, face.height * m_ratio.y));
^
FaceDetectorAndTracker.cpp:77:79: error: request for member ‘width’ in ‘face’, which is of non-class type ‘const int’
faces.push_back(cv::Rect(face.x * m_ratio.x, face.y * m_ratio.y, face.width * m_ratio.x, face.height * m_ratio.y));
^
FaceDetectorAndTracker.cpp:77:103: error: request for member ‘height’ in ‘face’, which is of non-class type ‘const int’
faces.push_back(cv::Rect(face.x * m_ratio.x, face.y * m_ratio.y, face.width * m_ratio.x, face.height * m_ratio.y));
^
FaceDetectorAndTracker.cpp: In member function ‘void FaceDetectorAndTracker::detect()’:
FaceDetectorAndTracker.cpp:86:5: error: ‘m_faceCascade’ was not declared in this scope
m_faceCascade->detectMultiScale(m_downscaledFrame, m_facesRects, 1.1, 3, 0,
^
FaceDetectorAndTracker.cpp:101:15: error: ‘face’ does not name a type
for (auto face : m_facesRects)
^
FaceDetectorAndTracker.cpp:111:5: error: expected ‘;’ before ‘m_faceRois’
m_faceRois.clear();
^
FaceDetectorAndTracker.cpp:111:22: error: could not convert ‘((FaceDetectorAndTracker)this)->FaceDetectorAndTracker::m_faceRois.std::vector<Tp, Alloc>::clear<cv::Rect, std::allocator<cv::Rect > >()’ from ‘void’ to ‘bool’
m_faceRois.clear();
^
FaceDetectorAndTracker.cpp:112:5: error: expected primary-expression before ‘for’
for (const auto& face : m_facesRects)
^
FaceDetectorAndTracker.cpp:112:5: error: expected ‘)’ before ‘for’
FaceDetectorAndTracker.cpp:112:22: error: ISO C++ forbids declaration of ‘face’ with no type [-fpermissive]
for (const auto& face : m_facesRects)
^
FaceDetectorAndTracker.cpp:112:29: error: range-based ‘for’ loops are not allowed in C++98 mode
for (const auto& face : m_facesRects)
^
FaceDetectorAndTracker.cpp:114:72: error: no matching function for call to ‘FaceDetectorAndTracker::doubleRectSize(const int&, cv::Size&)’
m_faceRois.push_back(doubleRectSize(face, m_downscaledFrameSize));
^
FaceDetectorAndTracker.cpp:114:72: note: candidate is:
In file included from FaceDetectorAndTracker.cpp:1:0:
FaceDetectorAndTracker.h:40:21: note: static cv::Rect FaceDetectorAndTracker::doubleRectSize(const Rect&, const Size&)
static cv::Rect doubleRectSize(const cv::Rect &rect, const cv::Size &frameSize);
^
FaceDetectorAndTracker.h:40:21: note:   no known conversion for argument 1 from ‘const int’ to ‘const Rect& {aka const cv::Rect_&}’
FaceDetectorAndTracker.cpp: In member function ‘void FaceDetectorAndTracker::track()’:
FaceDetectorAndTracker.cpp:133:21: error: ISO C++ forbids declaration of ‘roi’ with no type [-fpermissive]
const auto &roi = m_faceRois[i]; // roi
^
FaceDetectorAndTracker.cpp:133:39: error: invalid initialization of reference of type ‘const int&’ from expression of type ‘cv::Rect_’
const auto &roi = m_faceRois[i]; // roi
^
FaceDetectorAndTracker.cpp:136:55: error: invalid conversion from ‘int’ to ‘const cv::Range*’ [-fpermissive]
const cv::Mat &faceRoi = m_downscaledFrame(roi);
^
In file included from /usr/include/opencv2/core/core.hpp:4855:0,
from FaceDetectorAndTracker.h:3,
from FaceDetectorAndTracker.cpp:1:
/usr/include/opencv2/core/mat.hpp:381:12: error:   initializing argument 1 of ‘cv::Mat cv::Mat::operator()(const cv::Range*) const’ [-fpermissive]
inline Mat Mat::operator()(const Range* ranges) const
^
FaceDetectorAndTracker.cpp:137:9: error: ‘m_faceCascade’ was not declared in this scope
m_faceCascade->detectMultiScale(faceRoi, m_tmpFacesRect, 1.1, 3, 0,
^
FaceDetectorAndTracker.cpp:138:26: error: request for member ‘width’ in ‘roi’, which is of non-class type ‘const int’
cv::Size(roi.width * 4 / 10, roi.height * 4 / 10),
^
FaceDetectorAndTracker.cpp:138:46: error: request for member ‘height’ in ‘roi’, which is of non-class type ‘const int’
cv::Size(roi.width * 4 / 10, roi.height * 4 / 10),
^
FaceDetectorAndTracker.cpp:139:26: error: request for member ‘width’ in ‘roi’, which is of non-class type ‘const int’
cv::Size(roi.width * 6 / 10, roi.width * 6 / 10));
^
FaceDetectorAndTracker.cpp:139:46: error: request for member ‘width’ in ‘roi’, which is of non-class type ‘const int’
cv::Size(roi.width * 6 / 10, roi.width * 6 / 10));
^
FaceDetectorAndTracker.cpp:163:48: error: request for member ‘x’ in ‘roi’, which is of non-class type ‘const int’
m_facesRects[i].x = minLoc.x + roi.x - m_faceTemplates[i].cols / 2;
^
FaceDetectorAndTracker.cpp:164:48: error: request for member ‘y’ in ‘roi’, which is of non-class type ‘const int’
m_facesRects[i].y = minLoc.y + roi.y - m_faceTemplates[i].rows / 2;
^
FaceDetectorAndTracker.cpp:186:38: error: request for member ‘x’ in ‘roi’, which is of non-class type ‘const int’
m_facesRects[i].x += roi.x;
^
FaceDetectorAndTracker.cpp:187:38: error: request for member ‘y’ in ‘roi’, which is of non-class type ‘const int’
m_facesRects[i].y += roi.y;
^
In file included from FaceSwapper.cpp:1:0:
FaceSwapper.h:11:25: fatal error: dlib/opencv.h: No such file or directory
#include <dlib/opencv.h>
^
compilation terminated.
In file included from main.cpp:4:0:
FaceSwapper.h:11:25: fatal error: dlib/opencv.h: No such file or directory
#include <dlib/opencv.h>
^

Very interested in this project and would like little more guidance as well on setup.

There's more instructions in the README now


Support submitting multiple transactions at once

When submitting multiple transactions at once, there's a risk that same coins will be used in more than one tx which will cause some of them to fail.
We should add support and tests for this.

Closing this in favor of the accidental duplicate: #194


Support for PolyData?

Would you please consider adding PolyData support? It would be greatly appreciated. I use WriteVTK to save results from particle simulations and then view in paraview. Sometimes I would like to load a simulation state or do some complicated post-processing.
I have only bunch of points with point data. Actually, your package works when I add
  elseif file_type == "PolyData"
    piece = root[file_type][1]["Piece"][1]
    n_points = parse(Int, attribute(piece, "NumberOfPoints", required=true))
    n_cells = 0
  end

after line 140. But by default it throws assertion error.

Would you mind making a PR with the addition and some tests?

Maybe. But I dont understand vtk very much and the fix which works for me is for sure extremely not robust.
What do you imagine as tests?

A nice test would be to use some different ways to write different forms of PolyData via WriteVTK.jl, read them back in, and check whether you get what you wrote.

Hi. I  made a pull request with the addition and a test.


[SUPPORT] Exception when using MERGE INTO

I'm trying to use merge into and perform partial update on the target data but getting the following error:
java.lang.UnsupportedOperationException: MERGE INTO TABLE is not supported temporarily.
  at org.apache.spark.sql.execution.SparkStrategies$BasicOperators$.apply(SparkStrategies.scala:718)
  at org.apache.spark.sql.catalyst.planning.QueryPlanner.$anonfun$plan$1(QueryPlanner.scala:63)
  at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
  at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
  at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:491)
  at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:93)
  at org.apache.spark.sql.execution.SparkStrategies.plan(SparkStrategies.scala:67)
  at org.apache.spark.sql.catalyst.planning.QueryPlanner.$anonfun$plan$3(QueryPlanner.scala:78)

Steps to reproduce:

Load the target table

 val df = Seq(("1","neo","2023-08-04 12:00:00","2023-08-04 12:00:00","2023-08-04")).toDF("emp_id", "emp_name", "log_ts", "load_ts", "log_dt")

df.select(col("emp_id").cast("int"),col("emp_name").cast("string"),col("log_ts").cast("timestamp"),col("load_ts").cast("timestamp"),col("log_dt").cast("date"))

res0.write.format("hudi")
.option("hoodie.payload.ordering.field", "load_ts")
.option("hoodie.datasource.write.recordkey.field", "emp_id")
.option("hoodie.datasource.write.partitionpath.field", "log_dt")
.option("hoodie.index.type","GLOBAL_SIMPLE")
.option("hoodie.table.name", "hudi_test")
.option("hoodie.simple.index.update.partition.path", "false")
.option("hoodie.datasource.write.precombine.field", "load_ts")
.option("hoodie.datasource.write.payload.class","org.apache.hudi.common.model.PartialUpdateAvroPayload")
.option("hoodie.datasource.write.reconcile.schema","true")
.option("hoodie.schema.on.read.enable","true")
.option("hoodie.datasource.write.hive_style_partitioning", "true")
.option("hoodie.datasource.write.row.writer.enable","false")
.option("hoodie.datasource.hive_sync.enable","true")
.option("hoodie.datasource.hive_sync.database","pharpan")
.option("hoodie.datasource.hive_sync.table", "hudi_test")
.option("hoodie.datasource.hive_sync.partition_fields", "log_dt")
.option("hoodie.datasource.hive_sync.ignore_exceptions", "true")
.option("hoodie.datasource.hive_sync.mode", "hms")
.option("hoodie.datasource.hive_sync.use_jdbc", "false")
.option("hoodie.datasource.write.operation","upsert")
.mode("append")
.save("gs://sample_bucket/hudi_sample_output_data")

Load the incremental data

val df2 = Seq(("1","neo","2023-08-05 14:00:00","2023-08-04 12:00:00","2023-08-05"),("2","trinity","2023-08-05 14:00:00","2023-08-05 15:00:00","2023-08-05")).toDF("emp_id", "emp_name", "log_ts","load_ts","log_dt")

df2.select(col("emp_id").cast("int"),col("emp_name").cast("string"),col("log_ts").cast("timestamp"),col("load_ts").cast("timestamp"),col("log_dt").cast("date"))

res2.createOrReplaceTempView("incremental_data")

Perform merge

val sqlPartialUpdate =
           s"""
             | merge into pharpan.hudi_test as target
             | using (
            |   select * from incremental_data
            | ) source
            | on  target.emp_id = source.emp_id
            | when matched then
            |   update set target.log_ts = source.log_ts, target.log_dt = source.log_dt
            | when not matched then insert *
            """.stripMargin

spark.sql(sqlPartialUpdate)

Hudi verison: 0.13.1
Using "org.apache.hudi.common.model.PartialUpdateAvroPayload" for partial update.
Can someone please help in resolving this error? Also, please share the documentation on using MERGE INTO if I'm using it in the wrong way.

Hi, one thing I notice is that you are updating the partitionpath field value target.log_dt = source.log_dt. This may not be the cause of the exception, but also having "hoodie.simple.index.update.partition.path", "false" could result in some unknown behavior.
How are you running spark? What version of spark are you using?

As of now, I'm using spark-shell to test but eventually move to dataproc spark-submit. The spark version is 3.1.2

Can you check that the configs:
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension

are set and spelled correctly?
Also can you make sure that the spark jar is using scala 2.12?

@jonvex by adding the HoodieSparkSessionExtension config merge is working. However, I'm seeing three records instead of two records even though I changed this property to true "hoodie.simple.index.update.partition.path", "true" . Can you please suggest what else might be the issue?

@praneethh I tried the similar stuff using spark-sql. It gave me two records only. below if the code I used -
CREATE TABLE issue_9469_23 USING HUDI
PARTITIONED BY(log_dt)
tblproperties (
  "hoodie.payload.ordering.field"="load_ts",
  "hoodie.datasource.write.recordkey.field"="emp_id",
  "hoodie.datasource.write.partitionpath.field"="log_dt",
  "hoodie.datasource.write.precombine.field"="load_ts",
  "hoodie.index.type"="GLOBAL_SIMPLE",
  "hoodie.simple.index.update.partition.path"="true",
  "hoodie.datasource.write.payload.class"="org.apache.hudi.common.model.PartialUpdateAvroPayload",
  "hoodie.datasource.write.reconcile.schema"="true",
  "hoodie.schema.on.read.enable"="true",
  "hoodie.datasource.write.hive_style_partitioning"="true",
  "hoodie.datasource.write.row.writer.enable"="false"
) AS SELECT 1 as emp_id, 'neo' as emp_name, cast('2023-08-04 12:00:00' as timestamp) as log_ts, cast('2023-08-04 12:00:00' as timestamp) as load_ts, cast('2023-08-04' as date) as log_dt;

merge into issue_9469_23 as target using ( SELECT 1 as emp_id, 'neo_1' as emp_name, cast('2023-08-05 12:00:00' as timestamp) as log_ts, cast('2023-08-05 12:00:00' as timestamp) as load_ts, cast('2023-08-05' as date) as log_dt
                UNION
                SELECT 2 as emp_id, 'trinity' as emp_name, cast('2023-08-05 14:00:00' as timestamp) as log_ts, cast('2023-08-05 15:00:00' as timestamp) as load_ts, cast('2023-08-05' as date) as log_dt
  ) source on  target.emp_id = source.emp_id when matched then
  update set target.log_ts = source.load_ts, target.log_ts = source.log_dt, target.load_ts = source.load_ts
  when not matched then insert *;

@ad1happy2go still getting three records as output. I'm using the same properties you have used and additionally hive_sync properties. Also not creating the table manually, it is created by the hudi from the first run.

@praneethh I am able to reproduce the issue. Below is the code I used -
val tableName = "issue_9469_53"
val path = s"file:///tmp/${tableName}"
val df = Seq(("1","neo","1","1","2023-08-04")).toDF("emp_id", "emp_name", "log_ts", "load_ts", "log_dt")

df.select(col("emp_id").cast("int"),col("emp_name").cast("string"),col("log_ts").cast("int"),col("load_ts").cast("int"),col("log_dt").cast("date")).write.format("hudi")
.option("hoodie.datasource.write.recordkey.field", "emp_id")
.option("hoodie.datasource.write.partitionpath.field", "log_dt")
.option("hoodie.index.type","GLOBAL_SIMPLE")
.option("hoodie.table.name", tableName)
.option("hoodie.simple.index.update.partition.path", "false")
.option("hoodie.datasource.write.precombine.field", "load_ts")
.option("hoodie.datasource.hive_sync.enable","true")
.option("hoodie.datasource.hive_sync.database","default")
.option("hoodie.datasource.hive_sync.table", tableName)
.option("hoodie.datasource.hive_sync.partition_fields", "log_dt")
.option("hoodie.datasource.hive_sync.ignore_exceptions", "true")
.option("hoodie.datasource.hive_sync.mode", "hms")
.option("hoodie.datasource.hive_sync.use_jdbc", "false")
.option("hoodie.datasource.write.operation","upsert")
.mode("append")
.save(path)

val df2 = Seq(("1","neo","2","2","2023-08-05"),("2","trinity","2","2","2023-08-05")).toDF("emp_id", "emp_name", "log_ts","load_ts","log_dt")

val df3 = df2.select(col("emp_id").cast("int"),col("emp_name").cast("string"),col("log_ts").cast("int"),col("load_ts").cast("int"),col("log_dt").cast("date"))

df3.createOrReplaceTempView("incremental_data")

val sqlPartialUpdate =
           s"""
             | merge into ${tableName} as target
             | using (
            |   select * from incremental_data
            | ) source
            | on  target.emp_id = source.emp_id
            | when matched then
            |   update set target.log_ts = source.log_ts, target.log_dt = source.log_dt, target.load_ts = source.load_ts
            | when not matched then insert *
            """.stripMargin

spark.sql(sqlPartialUpdate)

Created the JIRA (https://issues.apache.org/jira/browse/HUDI-6737) for the same and will be working on it.

Hi @ad1happy2go wanted to check if there is any update on the above Jira. Thanks

@praneethh We are working on this. We will update soon.

@jonvex Can you post latest update on this?

@praneethh I have checked, and it appears that this was placed on the back burner. We will make it a priority this week. I apologize for any delay.

@ad1happy2go Thanks for the update.

@jonvex @ad1happy2go Circling back on this issue to check if there is any update. Sorry to bother on this multiple times.

Hi, really sorry we keep delaying this.

@praneethh We were facing some of the issues when you tried to fix it. Current code actually don't support partition path update with MERGE INTO. We will provide more update on this soon. Sorry for all the delay.

@codope @ad1happy2go @jonvex Hi Team, any update on the above issue?or is there any alternative solution?


[SPARK-23886][SS] Update query status for ContinuousExecution

What changes were proposed in this pull request?
Added query status updates to ContinuousExecution.
How was this patch tested?
Existing unit tests + added ContinuousQueryStatusAndProgressSuite.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Test build #99056 has started for PR 23095 at commit 5b743ab.

cc @jose-torres @HeartSaVioR @attilapiros

Test build #99056 has finished for PR 23095 at commit 5b743ab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99056/
Test PASSed.

Test build #99071 has started for PR 23095 at commit 44b3c91.

Test build #99071 has finished for PR 23095 at commit 44b3c91.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99071/
Test PASSed.

Merged build finished. Test PASSed.

Test build #99192 has started for PR 23095 at commit 86539d0.

Test build #99192 has finished for PR 23095 at commit 86539d0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99192/
Test PASSed.

cc @tdas @zsxwing

cc @vanzin

Looks good. I'd have chosen a shorter test name, but no biggie.
Merging to master.


Avoid competing swagger generation during test build

The previous implementation caused the concurrent builds for various frameworks to compete in writing to the output swagger.json file, causing exceptions during builds (since it happened in csproj).
In my local development I often failed to rebuild the full solution because some projects failed to build for this reason. It has also happened in CI (https://github.com/domaindrivendev/Swashbuckle.AspNetCore/pull/2886).
Note that the NSwag source code generator in NswagClientExample still outputs everything in one spot for all frameworks, but I haven't noticed failures from that. Maybe it's just less likely to write concurrently. I left it as is.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 89.87%. Comparing base (d793865) to head (11e1a0e).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##           master    #2897   +/-   ##
=======================================
  Coverage   89.87%   89.87%
=======================================
  Files          72       72
  Lines        2706     2706
  Branches      428      428
=======================================
  Hits         2432     2432
  Misses        274      274

Flag
Coverage Δ

Linux
89.87% <ø> (ø)

Windows
?

macOS
89.87% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Do you think this will fix that flaky test from yesterday? If so, can you unskip it?

Just re-running the last build a few times to be sure...


What format is the PEM supposed to be in?

What PKSC flavor is it meant to be in?
I've tried a PFX, I've tried a PEM of the private key and a PEM of the cert. They all wind up throwing an error:
root@cmz420:/opt/cole# azurectl compute vm regions
ERROR: Unexpected error:
Traceback (most recent call last):
  File "/usr/local/bin/azurectl", line 11, in <module>
    load_entry_point('azurectl==2.1.1', 'console_scripts', 'azurectl')()
  File "/usr/local/lib/python2.7/dist-packages/azurectl/azurectl.py", line 45, in main
    App()
  File "/usr/local/lib/python2.7/dist-packages/azurectl/app.py", line 31, in __init__
    task_class().process()
  File "/usr/local/lib/python2.7/dist-packages/azurectl/commands/compute_vm.py", line 135, in process
    self.__list_locations()
  File "/usr/local/lib/python2.7/dist-packages/azurectl/commands/compute_vm.py", line 165, in __list_locations
    self.result.add('regions', self.account.locations('PersistentVMRole'))
  File "/usr/local/lib/python2.7/dist-packages/azurectl/account/service.py", line 130, in locations
    for location in service.list_locations():
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementservice.py", line 1131, in list_locations
    Locations)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 365, in _perform_get
    response = self.perform_get(path, x_ms_version)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 175, in perform_get
    response = self._perform_request(request)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 337, in _perform_request
    resp = self._filter(request)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/_http/httpclient.py", line 183, in perform_request
    self.send_request_body(connection, request.body)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/_http/httpclient.py", line 142, in send_request_body
    connection.send(None)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/_http/requestsclient.py", line 81, in send
    self.response = self.session.request(self.method, self.uri, data=request_body, headers=self.headers, timeout=self.timeout)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 596, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 497, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL] PEM lib (_ssl.c:2603)

I expected, for example, to have to specify a client cert and the client private key...

openssl pkcs12 -in file.pfx -out file.pem -nodes did the trick, now I get an auth error but I assume that's a different issue.


ARC: Disable TLS for some configurations

Disable Thread Local Storage for some configurations of ARC architecture. For cores with more then one RGF_NUM_BANKS the parameter is disabled by-default because banks syncronization requires significant time, and it slows down performance.

@evgeniy-paltsev,  @ruuddw,  could you please take a look?


dosample fails when uploading split file

Uploading the split file splits.seq by dosample.py fails with
Traceback (most recent call last):
  File "dosample.py", line 31, in <module>
    main()
  File "dosample.py", line 27, in main
    run_sample_job()
  File "dosample.py", line 23, in run_sample_job
    key = job.fs.make_s3_key(job.options.splitfile)
AttributeError: 'SampleCDXJob' object has no attribute 'fs'

Seen in June and July crawl on EMR. Not reproducible on Hadoop / CDH.


How to configure, backup, automate, etc?

As the DevOps side of a development team that is looking to incorporate this into their project I have a few questions.
Is there a way to have a config file outside of the data directory so that we can configure settings such as data directory, log file location, authentication, replication, etc. using our config management tools such as puppet or chef?
How do we back this up? Is there a way to dump the data to a backup file or do we backup a file in the data directory? Once we have a backup, how do we restore it? Is a backup/restore how we migrate the data from one server to another?
If we have a primary and a follower and the primary goes down can we have the secondary become primary and if the old primary comes back up, it becomes secondary?
I think that I figured out a way to run this as a systemd service using the command line switches but would like to have it running from a config file if possible.
Thanks.

Hi Cory,

Is there a way to have a config file outside of the data directory so that we can configure settings such as data directory, log file location, authentication, replication, etc. using our config management tools such as puppet or chef?

There isn't currently an option to load a config file from a different directory. Right now everything resides in the data directory. The log is written to stderr.

How do we back this up? Is there a way to dump the data to a backup file or do we backup a file in the data directory? Once we have a backup, how do we restore it? Is a backup/restore how we migrate the data from one server to another?

The data/appendonly file contains all of the data. It's safe to copy this file at any time for backup. I recommend gzipping ahead of time. To restore, just put the appendonly file back into place and start the server. To migrate, you can copy the file to the new server.

If we have a primary and a follower and the primary goes down can we have the secondary become primary and if the old primary comes back up, it becomes secondary?

No, the follower will remain a follower (read-only) until it receives the command LEADER none. Unfortunately there's no builtin failover feature at this time.
I hope this answers your questions, and please let me know if you have any more.
Thanks for your interest in Tile38!

@tidwall maybe we should add some of this into doc's?

@m1ome agreed. I'll keep this open for now.

@tidwall awesome.

is there any memory policy when reached the memory bottleneck?

Is there any improvement?


Update the game over message

Update the game over message so people know how to play again!

this is not hard


Get Segmentation Fault when using libxsmm_bmmdispatch

I build the dynamic library using the following:
wget https://github.com/hfp/libxsmm/archive/1.14.tar.gz
tar xvf 1.14.tar.gz
cd libxsmm-1.14/
make STATIC=0

and the following code gives an segmentation fault error:
#include <libxsmm.h>
#include <vector>

int main(int argc, char* argv[]) {
  float alpha = 1.0;
  float beta = 1.0;
  typedef libxsmm_bfloat16 value_type;
  int batchsize = 1, m = 1, n = 32, k = 32;
  std::vector<value_type> a(batchsize*m*k), b(batchsize*k*n), c(m*n, 0);
  libxsmm_bmmfunction kernel;
  kernel = libxsmm_bmmdispatch(m, n, k, &m, &n, &k, &alpha, &beta, NULL, NULL);
  assert(kernel);
  for (int i = 0; i < batchsize; ++i) { /* initialize input */
    a[i*m*k] = static_cast<value_type>(1);
    b[i*k*n] = static_cast<value_type>(7);
  }
  /* kernel multiplies and accumulates matrix products: C += Ai * Bi */
  for (int i = 0; i < batchsize; ++i) kernel(&a[i*m*k], &b[i*k*n], &c[0]);
}

my bazel BUILD file is as following:
cc_library(
  name = "xsmm_avx",
  srcs = glob(["lib/*.so"]),
  hdrs = glob(
      [
          # trigger rebuild if template changed
          "src/template/*.c",
          "src/*.h",
          "include/*.h",
      ]
  ) + [
      # source files included internally
      "src/libxsmm_hash.c",
  ],
  includes = [
      "include",
      "src",
      "src/template",
  ],
  linkopts = ["-lpthread"],
  visibility = ["//visibility:public"],
)

Would you please tell me where I am wrong ?

Running a debug-build of this case (or using LIBXSMM_VERBOSE=1 environment variable), unveils the following message:
LIBXSMM ERROR (libxsmm_generator_gemm_kernel): unknown architecture or unsupported precision (error #90004)!

Granted, the error message could be improved. Anyhow, the prerequisite for running BFloat16 is "CPX". We do not have fall-back code for BFloat16-based GEMM on earlier hardware (and CPX is not publicly available at this point in time).

( @alheinecke may want to comment on the above note )

No the bug must be something else. There is fallback to AVX512. So the null pointer will only come on non-AVX512 platform, but AVX512 is fine.
Of course k%2 needs to hold true due to the instructions.

@alheinecke I run a test on both an AVX512 supported machine and AVX256 supported machine, neither passes unfortunately.

For our own reference/documentation ("please ignore"), I dropped below content into LIBXSMM's root directory (file is called BUILD).
cc_library(
    name = "xsmm_avx",
    srcs = glob(["lib/*.so"]),
    hdrs = glob(
        [
            # trigger rebuild if template changed
            "src/template/*.c",
            "src/*.h",
            "include/*.h",
        ]
    ) + [
        # source files included internally
        "src/libxsmm_hash.c",
    ],
    includes = [
        "include",
        "src",
        "src/template",
    ],
    linkopts = [
        "-lpthread",
        "-fopenmp",
    ],
    visibility = ["//visibility:public"],
)

cc_binary(
    name = "simple",
    srcs = ["simple.cpp"],
    includes = [
      "include",
    ],
    deps = [
        "//:xsmm_avx"
    ],
)

The above given source code was called simple.cpp and dropped into the root folder as well. This depends on LIBXSMM built per make STATIC=0 (DBG=1 may be advisable). Finally, I used Bazelisk to build the example (bazelisk build //:simple).

It seems you use a very old version (1.14), please use the latest master. Again, we only support AVX512.

Thank you @alheinecke .
In my opinion, in matrix mulplication,
C = alpha * A * B + beta * b,
where A has a dimension (m, n), B has a dimension (n, k), and lda is m, ldb is n, ldc is k.
Am I right ?
And if I have a matrix with dimension 512 * 512, and a vector with dimension 512 * 1, is it possible that libxsmm show its power ? Or I will only get 1 / 512 of peak performance ?
Thank you !

No, LIBXSMM is following BLAS standard and is col-major that means in C:
A[K][M] , B[N][K] and C[N][M] -> C[N][M] = alpha \sum_k A[k][M] * B[N][k] + betaC[N][M]  \forall N,M
that means we have following requirements for LDs: LDA>=M, LDB>=K and LDC>=M
When you multiply a matrix with a vector, we are not bad, but this is not the focus. The focus is matrix X matrix. For (dense) matrix * vector even simply C code will do as this is a read memory bandwidth bound problem.
Also we are focusing on sizes up to 128. Above one needs to assemble a custom operation using our fast micro kernels.

@alheinecke Thank you for you reply.
I am glad that I was wrong so I can go on a right way more quickly.
You said that multiply a matrix with a vector is not the focus, but actually I think we have needs for them. Have you heard this repository ? They have a "packed" concept and they will pack the matrix before computing and do the matrix vector muplication job for a lot of times which save the packing overhead and improve the inference speed.
My question is that, do you get different focus ? or more specificly, wil libxsmm act  as well as that repository ?

yes, we know FBGEMM. However, my concern/question is around matrix*vector. Even packing the matrix, you still have N^2 memory and N^2 compute operations and therefore you are still BW-bound.
Can you explain a bit more if you are looking into computer vision or natural language processing?
On how to use small matmul of LIBXSMM in deep learning I recommend the following reading:
https://arxiv.org/abs/1808.05567
https://arxiv.org/abs/1906.06440

Thank you for your reply @alheinecke ,
I am looking into speech generation problems. Most of them are autoregresive models.
I will generate audios, with sample rate like 48k Hz. It means that when 1 second audio is generated, 24000 times of all matrix mulplications needs to be computed.
Most of the matrix mulplications come from rnn cells, like grus and dense layers. Due to the limit of rnns, the mulplication has to be computed between a vector with size like (1, 512) and (512, 512).
Is it possible that FBGEMM performs as well as LIBXSMM ?

all of the libraries will be the same as they operation cannot ran run fast as it's memory bandwidth bound and not compute bound. I would suggest you look for a potentially batching approach in your application.

@hfp The segmentation fault error is disappeared using the following code, but the result is not right.
#include <libxsmm.h>
#include <stdint.h>
#include <iostream>
#include <vector>

int main(int argc, char* argv[]) {
  float alpha = 1.0;
  float beta = 1.0;
  typedef libxsmm_bfloat16 value_type;
  int batchsize = 1, n = 32, m = 32, k = 32;

  std::vector<value_type> a(batchsize * m * k), b(batchsize * k * n);
  std::vector<float> c(m * n);
  for (int i = 0; i < 32 * 32; i++) {
    a[i] = 1;
    b[i] = 1;
    c[i] = 0;
  }
  libxsmm_bsmmfunction kernel;
  kernel = libxsmm_bsmmdispatch(m, n, k, &m, &k, &m, &alpha, &beta, NULL, NULL);

  for (int i = 0; i < batchsize; ++i) kernel(a.data(), b.data(), c.data());
  std::cout << "c" << std::endl;
  std::cout << c[0] << std::endl;

  std::cout << "a" << std::endl;
  std::cout << a[0] << std::endl;
  std::cout << "b" << std::endl;
  std::cout << b[0] << std::endl;
}

build file is
cc_binary(
    name = "test_libxsmm",
    srcs = [
        "test_libxsmm.cc",
    ],
    copts = [
        "-fopenmp",
        "-D_OPENMP",
        "-m64",
        "-mfma",
        "-mavx512f",
        "-mavx512bw",
        "-mavx512dq",
        "-mavx512vl",
        "-masm=intel",
    ],
    linkopts = [
        "-fopenmp",
        "-lm",
    ],
    deps = [
        "@libxsmm_archive//:xsmm_avx",
    ],
)

the output is
LIBXSMM WARNING: highly optimized non-JIT code paths are limited to "hsw"!

LIBXSMM WARNING: timer is maybe not cycle-accurate!
c
0
a
1
b
1

LIBXSMM_VERSION: master-sc20_paper-95<PHONE_NUMBER>)
CLX/SP      TRY    JIT    STA    COL
   0..13      0      0      0      0
  14..23      0      0      0      0
  24..64      1      1      0      0
Registry and code: 10 MB + 20 KB

where c[0] should be 32.
Can you point out where I am wrong ? Thank you !

I believe “1” is not 1.0 in terms of Bfloat16. You need a conversion or use a ready to use copy-in routine. That’s because C/C++ does not yet support any FP type smaller than SP. You may find quantization code in our bf16 sample code or find some ready to use routine perhaps in libxsmm_dnn.h.

For C++ we can perhaps improve with a type that does the conversion by implementing a suitable c’tor/d’tor.

Thank you !

yes you need to use this function: https://github.com/hfp/libxsmm/blob/master/src/libxsmm_dnn.c#L714
or the union directly in your code without RNE rounding, but truncation:
libxsmm_bfloat16_hp t;
t.f = (float)1;
mybf16data[i] = t.i[1];


Set custom min max axis values

This chart.js code is not working here
ticks: {
                beginAtZero: true,
                min: 0,
                max: 100,
              }

You could start by investigating here: https://github.com/react-tools/react-charts/blob/master/src/utils/buildAxis.linear.js#L29
or here: https://github.com/react-tools/react-charts/blob/master/src/components/AxisLinear.js#L43

Also, please comment with a codesandbox.io demonstrating the problem. If we can't replicate it, then we can't fix it properly.

@tannerlinsley i think I got what I was looking for. Thanks :)

how to provide min value as 0 to axis?
I am not able to get output on codesandbox.io but the code is at https://codesandbox.io/s/bar-chat-min-axis-not-working-example-t0qy5?file=/src/App.js

Thank You

For future self:
To make the bar chart start at zero:
const secondaryAxes = React.useMemo(() => [
    {
        getValue: datum => datum.value,
        elementType: 'bar',
        min: 0
    },
], []);


Navbar_As a user, I can navigate to pages on the public facing website (home, projects, about)

Child of #106

merge into #106


0.13.1 version cameraview has stretched preview

Is this a bug report?
(yes)

version 0.13.0 works fine
OS 6.0.0

I have the same problem, but only in a specific device with Android 4.4.2.
Only the preview is stretched, the result image is fine.

Same here: Xiaomi Redmi Note 3, Android 5.02 LRX22G, MIUI 8.5

Ok drafted a 0.13.2 milestone and added this to it, we will see what we can do.

thanks, and also can anyone fix flash issue? when method mode is steel flashlight does not works #266

Added @developer--

I am updated from 0.13.12 - see the same problem. Meizu m5 note

Same error, preview is distorted when screen launch with portrait and then change to landscape.

FYI, I had the same issue and reverted back to 0.13.0 as recommended by this thread to fix it. Will stay on this version until 1.0.0 is released.

Hoping to PR 1.0.0 this weekend. Will keep you posted.

@littleariel I thought things were okay with 13 but i'm seeing some aspect ratio issues on some tablets, nexus 7/10 for example

Should be resolved in #318 can you please test that out and I will reOpen this if you comment that it is still an issue.

Hi,
I have a similar problem also with 0.13.0. In landscape on Samsung galaxy S2 Tablet with this option:
<com.wonderkiln.camerakit.CameraView
android:id="@+id/camerakit_view"
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:adjustViewBounds="true"
android:layout_centerInParent="true"
android:screenOrientation="portrait"
android:keepScreenOn="true"
/>
I have the preview stretched. Am I making any mistakes?
Thanks

How to resolve issue?


Parameter 'props' implicitly has an 'any' type

After including your project, a typecheck fails (even with --skipLibCheck):
node_modules/react-native-circular-progress-indicator/src/circularProgress/index.tsx:184:31 - error TS7006: Parameter 'props' implicitly has an 'any' type.

184 export const dynamicStyles = (props) => {
                                  ~~~~~
Found 1 error.

You can either fix your implicit anys or compile your project to a type definition and commonjs source and publish that instead of the plain typescript source.

Thanks for reporting this. The lib is now published as a commonjs source and type definitions with the help of react-native-builder-bob starting from v4.0.0.


Creating fbterm terminal information

Creating new terminal definition for framebuffter terminal emulator fbterm:
cd terminfo
go build mkinfo.go
TERM=fbterm ./mkinfo -go term_fbterm.go

Does this need to be a .go file?  Maybe we can just create the JSON format instead?  It seems like fbterm is probably a bit of an edge case?

Does this need to be a .go file? Maybe we can just create the JSON format instead? It seems like fbterm is probably a bit of an edge case?

Sure, I also tried that but failed to include the file afterwards. Is there some magic involved?

I'm closing this for now.  Please resubmit later with the new format if you still need it.


Features that are transitively enabled by default are not rendered as "(enabled by default)"

With
[features]
default=["qqq"]

## Qqq
qqq=["www"]

## Www
www=[]

I expect

qqq (enabled by default) — Qqq
www (enabled by default) — Www

, but get

qqq (enabled by default) — Qqq
www — Www

.

Yes, i think you're right and it should probably have (enabled by default)
or maybe it should have (enabled through the "qqq" feature) ?

Or maybe "(enabled by default, enabled through the "qqq" feature)" where document-features exposes relationship between all documented features (even if it is going through a non-documented feature) even when they are not enabled by default.


Refactor spatial analysis (old)

Created a new spatial refactor branch with all the changes from the previous, this time branched directly off master. This is to prevent its accidental deletion from the merging of other branches.

I can delete this right?

Hi Noah, you can delete this branch.


transform translate-y-1/2 inot work ?

for example(https://snack.expo.dev/UFdZPhvJF)

React Native does not allow percentage values for translate transforms. There is a poly-fill for this in the upcoming v4


Majd 3rd Pull Request

Base.html as the base of the Webapp, Still issues with the WTForms forms, can't render them. Please look into the issue

I will look into it. Thank you.


Document break to Contains if the SQL Server compatibility level is too low

Users running on really old (unsupported) versions of SQL Server - as well as those who have set the compatibility level to a low value - will break when upgrading to EF Core 8.0, since the translation of Contains now relies on OPENJSON (see https://github.com/dotnet/efcore/issues/31323 for an example). We should add a breaking change note for this.
The same happens if using SQLite versions older than 3.38.0. This is more rare as usually people use the bundled SQLite, but it could happen.

Regarding "unsupported", SQL Server 2014 and 2016 both have extended support for a few more years yet. I get that support here probably can't mean guaranteeing it continues to work with external tools & libraries, but given EF is also Microsoft, I just wondered if there is a full end-to-end customer support experience to consider here?
To be clear, I don't want EF to be held back by SQL Server's differing support cycle, just seeing if you have a good response prepared for potential grumbles 😀

@stevendarby see https://github.com/dotnet/efcore/issues/31362#issuecomment-1652920760

Brilliant, so they are supported?

Documentation: #4428

@roji Did you file an issue to check the compat level the first time we open a connection?

No, I'm still thinking about it :)


Release 4.7.8

No need for early release, since no one has reported any issues.

But nothing wrong with updating package-release-notes, but no version bump needed.


Module not working with disabled WinRM Basic Authentication

Hi,
customer of mine has disabled WinRM Basic Authentication on their clients via Group Policy, therefore trying to do a Connect-IPPSSession does not connect with error:
Connecting to Security & Compliance Center. Please logon in the new window
WARNING: Please note that you can only use above 9 new EXO cmdlets (the one with *-EXO* naming pattern).You can't use other cmdlets as we couldn't establish a Remote PowerShell session as bas
ic auth is disabled in your client machine. To enable Basic Auth, please check instruction here https://docs.microsoft.com/en-us/powershell/exchange/exchange-online-powershell-v2?view=exchang
e-ps#prerequisites-for-the-exo-v2-module
Create Powershell Session is failed using OAuth
At C:\Program Files\WindowsPowerShell\Modules\ExchangeOnlineManagement\3.1.0\netFramework\ExchangeOnlineManagement.psm1:733 char:21
+                     throw $_.Exception;
+                     ~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [], Exception
    + FullyQualifiedErrorId : Create Powershell Session is failed using OAuth

Based on the current documentation (afaik) LearnWebSite - there is currently no way around this even with the V3 ExchangePowerShell Module.
Is there any way to run this compliance report without enabling basic authentication?
BG Christoph

Nevermind, we've solved the issue with the current Exchange PowerShell Module 3.2.0 - Preview4.
Upgrading to this version with:
Update-Module -Name ExchangeOnlineManagement -AllowPrerelease

or doing a plain install with:
Install-Module -Name ExchangeOnlineManagement -AllowPrerelease

and using (currently required as long as the module is not GA) the following command to connect:
Connect-IPPSSession -UseRPSSession:$false

Did the trick.
BG Christoph


Compile in OSX

remove VoronoiDiagramGenerator:: in VoronoiDiagramGenerator.h from the methods already in the class.
Also, this is a bit sad:
~/build/voronoi: make
/usr/local/bin/g++-4.8  -fopenmp -O2 -L/usr/local/lib/ -o voronoi_stippler picopng/picopng.o     stippler/bitmap.o stippler/stippler_api.o stippler/stippler.o stippler/VoronoiDiagramGenerator.o    voronoi/parse_arguments.o voronoi/voronoi.o -lboost_program_options
~/build/voronoi: ./voronoi_stippler -s 16000 -I corpus/klaymen.png -O test.svg
voronoi_stippler(77937) malloc: *** error for object 0x10365c570: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

This was fixed by 8ef7a04 which now defaults to using Clang in OS X along with better documentation of the build process in the readme.md file.


[SPARK-40276][CORE] Reduce the result size of RDD.takeOrdered

What changes were proposed in this pull request?
use Array instead of BoundedPriorityQueue to store intermediate results
Why are the changes needed?
1, encountered a case that RDD.takeOrdered fails due to Total size of serialized results of xxx tasks (... MiB) is bigger than spark.driver.
2, performance improvement:
bin/spark-shell --driver-memory=4G
Seq(10, 100, 1000, 10000, 50000, 100000).foreach { n => val start = System.currentTimeMillis; Seq.range(0, 10).foreach(_ => sc.range(0, 100000000, 1, 1000).top(n)); val duration = System.currentTimeMillis - start; println(s"n=$n, duration=$duration") }

duration
n=10
n=100
n=1,000
n=10,000
n=50,000
n=100,000

master
2,552
2,197
2,543
10,003
58,552
org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 763 tasks (1024.6 MiB) is bigger than spark.driver

this PR
2,556
2,138
2,196
7,371
33,903
66,895

this PR + treeReduce
9,160
9,748
9,728
11,441
17,216
24,728

it is strange that this PR + treeReduce turns out to be slowest when n is small, so still use reduce in this PR.
Does this PR introduce any user-facing change?
No
How was this patch tested?
added UT

Seems fine. Maybe could switch to treeReduce when the size is large, but no big deal

Merged into master, thank you all!


Cypress integration tests

This PR adds cypress integration tests.
The url can be edited in web/cypress/support/const.js
These tests are not checking data. It only ensure that functionalities are working correctly.
The goal is to keep easy to maintain tests to:

avoid regressions
reduce testing time spent on each review

You can run manually your test using:
make cypress

Or record your results and send them to the dashboard using:
npm run cypress:record

You will need a private CYPRESS_RECORD_KEY environment variable for that. Feel free to ask in PM.
Then embed the results of your branch using:
[![network-observability-console-plugin](https://img.shields.io/endpoint?url=https://dashboard.cypress.io/badge/simple/tjknpb/YOUR_BRANCH&style=flat)](https://dashboard.cypress.io/projects/tjknpb/runs)

@memodi you may be interested by this PR
This could be run in our CI / long test running to simulate activity in front end

Is there any way that we can integrate this in GitHub actions or Prow? Even if e.g. we need to mock some services.

Yes definitely. I'm digging on a way to run our plugin outside of cluster. For now I need to manage a lot of issues with dependencies.
If so then we will only need to mock loki queries with json-server for example :trophy:


Convert int to string with strconv package for performance

Signed-off-by: songjiayang<EMAIL_ADDRESS>

Hi! Thanks for the PR - out of interest, was this done with some kind of automated tooling?  What motivated the change?  Are there any measurable performance increases?

hi @tomwilkie , I just use vscode's global finding&replace to update codes :smile:
I do a benchmark,  using strconv package about 3x times performance improvement.
func BenchmarkInt2StringWithFmt(b *testing.B) {
	for i := 0; i < b.N; i++ {
		fmt.Sprintf("%d", 1024)
	}
}

func BenchmarkInt2StringWithStrconv(b *testing.B) {
	for i := 0; i < b.N; i++ {
		strconv.Itoa(1024)
	}
}

$ go test -bench=.
goos: darwin
goarch: amd64
pkg: gobench
BenchmarkInt2StringWithFmt-4       	15332882	        82.3 ns/op
BenchmarkInt2StringWithStrconv-4   	42170791	        29.2 ns/op
PASS
ok  	gobench	2.611s

Thanks @songjiayang! When I said "measurable performance increases", I meant with observable Prometheus behaviour, not microbenching the string conversion - its obviously quicker to use strconv :-)


Merging the patch from  Joeclinton1/google-images-download@c773e1c

this one fixed the "not downloading" problem: Joeclinton1/google-images-download@c773e1c
See https://github.com/garyfeng/google-images-download/commit/194cc47cc2bdfae73bc1205fa1ce2b00f1926bf2
I merged the JoeClinton "patch-1" branch into my main.

tested ok


Can't use the sdk

Is this supposed to have a .framework file? I can't find it anywhere. I'm following the instructions of referencing the API source code directly, but it won't show me the login screen (it's just empty).
Regards,
F

FYI,
I put the "-fno-objc-arc" flag for the sdk classes so I can build them in my ARC project.

@francoC we definitely recommend using the cocoapods approach to integrating the Live SDK to avoid that problem.


(Bug) InternetConnection

solution: we don't need to send this error to bagsnag

Will be tested after release on Prod

Checked on eWallet ver0.14
The issue with internet connection isn't reproduce.
The new issue with internet connection aren't received in bugsnag


added a "main" to packag.json to fix import issue

little fix to package.json to allow easy importing.

Hey @daiagi so sorry I missed this - thanks! Merged


Cannot call abstract method parent::options() error with Nova v1.3.1

Using Laravel Nova v1.3.1

@rickgoemans @phoenixg  I am not able to reproduce the error on Nova 1.3
Can you make sure you have the latest version of the repo and show me your filter class?

Here is my test class
<?php

namespace App\Nova\Filters;

use Illuminate\Database\Eloquent\Builder;
use Illuminate\Http\Request;
use App\Models\Campaign;

class ColumnFilter extends \philperusse\Filters\ColumnFilter
{
    public function apply( Request $request, $query, $value )
    {
        $args = collect($value)->values()->filter();
        return ($args->count()!==3) ?
            $query :
            $query->where(...$args->all());

    }

    public function options( Request $request )
    {
        return array_merge(parent::options($request), [
            'columns' => [
                'clicks'      => 'Clicks',
                'impressions' => 'Impressions',
                'spent'       => 'Spent',
                'amount'    => 'Revenues',
            ]
        ]);
    }
}

New release : https://github.com/philperusse/nova-column-filter/tree/v0.2.1


operator gitlab-operator-kubernetes (0.18.1)

Thanks for submitting your Operator. Please check the below list before you create your Pull Request.
New Submissions

[ ] Are you familiar with our contribution guidelines?
[ ] Have you packaged and deployed your Operator for Operator Framework?

[ ] Have you tested your Operator in all supported installation modes?
[ ] Have you considered whether you want to use semantic versioning order?
[ ] Is your submission signed?
[ ] Is operator icon set?

Updates to existing Operators

[ ] Did you create a ci.yaml file according to the update instructions?
[ ] Is your new CSV pointing to the previous version with the replaces property if you chose replaces-mode via the updateGraph property in ci.yaml?
[ ] Is your new CSV referenced in the appropriate channel defined in the package.yaml or annotations.yaml ?
[ ] Have you tested an update to your Operator when deployed via OLM?
[ ] Is your submission signed?

Your submission should not

[ ] Modify more than one operator
[ ] Modify an Operator you don't own
[ ] Rename an operator - please remove and add with a different name instead
[ ] Modify any files outside the above mentioned folders
[ ] Contain more than one commit. Please squash your commits.

Operator Description must contain (in order)

[ ] Description of the managed Application and where to find more information
[ ] Features and capabilities of your Operator and how to use it
[ ] Any manual steps about potential pre-requisites for using your Operator

Operator Metadata should contain

[ ] Human readable name and 1-liner description about your Operator
[ ] Valid category name1
[ ] One of the pre-defined capability levels2
[ ] Links to the maintainer, source code and documentation
[ ] Example templates for all Custom Resource Definitions intended to be used
[ ] A quadratic logo

Remember that you can preview your CSV here.
--
1 If you feel your Operator does not fit any of the pre-defined categories, file an issue against this repo and explain your need
2 For more information see here

/retest

/merge possible

/merge possible

/approve

The link above is broken, this seems to work:
https://k8s-operatorhub.github.io/community-operators/operator-ci-yaml/#reviewers

/merge possible


Generating label for custom dataset

Hello,
Thank you for sharing your work. I am currently trying to generate token label to a custom dataset for model lvvit_s, but I keep getting the loss close to 7 and the Accuracy 0 (not pre-trained and using 1 GPU in Google Colab). I also tried using the pre-trained model with --transfer  but got 0 in both Loss and Acc . What option should I use for a custom dataset?

Hello,
Thank you for sharing your work. I am currently trying to generate token label to a custom dataset for model lvvit_s, but I keep getting the loss close to 7 and the Accuracy 0 (not pre-trained and using 1 GPU in Google Colab). I also tried using the pre-trained model with --transfer but got 0 in both Loss and Acc . What option should I use for a custom dataset?

Are you using generate_label.py to generate label?
Large loss and zero accuracy usually mean you are not using the correct number of classes for the label generation. My current implementation is only tested when class number = 1000. I'm not sure if generate_label.py works fine when you apply it on custom dataset. You may check with this script and adapt it to your own dataset.

with --transfer but got 0 in both Loss and Acc

It's because if transfer flag is used in generate_label.py line 275 disable tracking of loss and accuracy.  But it not clear to me why you can't collect them during transfer.


backend/drm: use local DRM FD for wlr_rend

The new wlr_renderer_autocreate API is great for compositors, however
it causes some issues with DRM multi-GPU support.
A DRM child backend wants the compositor to use the parent GPU, so it
exposes the parent's DRM FD in get_drm_fd. However, in order to be able
to perform multi-GPU buffer copies, the child DRM backend still needs to
create a local renderer.
Use the new private wlr_renderer_autocreate_with_drm_fd function to
avoid creating a renderer for the parent GPU.
Fixes: e128e6c08dc0 ("render: drop egl parameters from wlr_renderer_autocreate")

This brings back external monitors with WLR_DRM_NO_MODIFIERS=1 Thanks!

This brings back external monitors with WLR_DRM_NO_MODIFIERS=1 Thanks!


Validation error when using maxbytes on ImageBlob

The issue
Validation error when using maxbytes on ImageBlob.

The stacktrace
jhipster-uml jhipster-jdl.jh
An error has occurred:
WrongValidationException
Error message:
The validation 'maxlength' isn't supported for the type 'ImageBlob'.
Stack trace:
Error
at Object. (/Volumes/Data/.nvm/versions/node/v4.4.3/lib/node_modules/jhipster-uml/lib/exceptions/wrong_validation_exception.js:7:38)
at Module._compile (module.js:409:26)
at Object.Module._extensions..js (module.js:416:10)
at Module.load (module.js:343:32)
at Function.Module._load (module.js:300:12)
at Module.require (module.js:353:17)
at require (internal/module.js:12:17)
at Object. (/Volumes/Data/.nvm/versions/node/v4.4.3/lib/node_modules/jhipster-uml/lib/editors/modelio_parser.js:10:32)
at Module._compile (module.js:409:26)
at Object.Module._extensions..js (module.js:416:10)

Steps to reproduce
Create Jhipster project with SQL type (I am using Postgres). Create JDL file with ImageBlob and maxbytes. Run Jhipster-UML

The XMI or JDL file
entity Make {
name String required,
logo ImageBlob maxbytes(1000000),
created ZonedDateTime required,
approved Boolean required
}

Versions

JHipster      3.0.0
JHipster UML    1.6.5

JHipster Version
jhipster-uml -v

From the stacktrace, you were using maxlength. Can you re-check?

That is what is odd, I saw the same thing, however nowhere in the JDL is maxlength used. As soon as I removed maxbytes, it ran.

The JDL grammar doesn't perform any kind of check, it's oly after the parsed JDL is received by JHipster UML that we check... I'll test the JDL.

Ok, there is a bug somewhere. I'll find it and fix it tonight or tomorrow evening. Thanks for reporting this, and sorry of the inconvenience.

No worries, thought I was crazy. Kept double checking my typing and if I had enough caffeine. Not holding me back right now just came across it. Cheers!

Hi, this is a JDL bug. I'll open an issue thread in the issue tracker and commit the fix.
The fix will be shipped in the next release (which may be available before the end of the week).


test: e2e - fix/tag failing tests (maintenance)

Allure Report
allure-report-publisher generated test report!

processReports: ✅ test report for e6dc85ee

passed
failed
skipped
flaky
total
result

Total
33
0
4
0
37
✅


Get-DbaDiskSpace - Script Analyzer Test failure

We're adding Pester tests and AppVeyor to dbatools, see https://github.com/sqlcollaborative/dbatools/wiki/AppVeyor-and-Pester for more detail.
The following rules were highlighted for this function:

RuleName
Severity
Line
Message

PSAvoidUsingPlainTextForPassword
Warning
128
Parameter '$SqlCredential' should use SecureString, otherwise this will expose sensitive information. See ConvertTo-SecureString for more information.

PSAvoidUsingWMICmdlet
Warning
182
File 'Get-DbaDiskSpace.ps1' uses WMI cmdlet. For PowerShell 3.0 and above, use CIM cmdlet which perform the same tasks as the WMI cmdlets. The CIM cmdlets comply with WS-Management (WSMan) standards and with the Common Information Model (CIM) standard, which enables the cmdlets to use the same techniques to manage Windows computers and those running other operating systems.

PSUseDeclaredVarsMoreThanAssignments
Warning
187
The variable 'query' is assigned but never used.

PSAvoidUsingWMICmdlet
Warning
189
File 'Get-DbaDiskSpace.ps1' uses WMI cmdlet. For PowerShell 3.0 and above, use CIM cmdlet which perform the same tasks as the WMI cmdlets. The CIM cmdlets comply with WS-Management (WSMan) standards and with the Common Information Model (CIM) standard, which enables the cmdlets to use the same techniques to manage Windows computers and those running other operating systems.

PSAvoidUsingWMICmdlet
Warning
210
File 'Get-DbaDiskSpace.ps1' uses WMI cmdlet. For PowerShell 3.0 and above, use CIM cmdlet which perform the same tasks as the WMI cmdlets. The CIM cmdlets comply with WS-Management (WSMan) standards and with the Common Information Model (CIM) standard, which enables the cmdlets to use the same techniques to manage Windows computers and those running other operating systems.

PSUseOutputTypeCorrectly
Information
350
The cmdlet 'Get-DbaDiskSpace' returns an object of type 'System.Collections.ArrayList' but this type is not declared in the OutputType attribute.

see https://github.com/sqlcollaborative/dbatools/issues/627 , aka "the issue that tracked them all"


[Core] aaz: Fix AAZBearerTokenCredentialPolicy to support MSIAuthenticationWrapper crendital

Related command

Description

Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature

This checklist is used to make sure that common guidelines for a pull request are followed.

[ ] The PR title and description has followed the guideline in Submitting Pull Requests.

[ ] I adhere to the Command Guidelines.

[ ] I adhere to the Error Handling Guidelines.

Core


Add information about component instance identity to keys documentation

For issue https://github.com/reactjs/reactjs.org/issues/2904.
Add information about component instance identity to keys documentation.
The current docs on lists and keys and reconciliation don't dig into the underlying reason why keys are helpful to provide component identity, which is helpful in other cases beyond improving list render performance.
Would welcome any feedback!

Closing as per comment in the issue.


[Question] How to deal with header variables like Authorization?

Hi, in graphql I have a cookie with my apollo-token and check for it in my context.
export default async ({ req, connection }) => {

  const authToken = req.get("Authorization")
  let user = authToken ? await tradeTokenForUser(authToken) : null

  return {
    user
  }
}

You wrote that we can setup a middleware in express for reading the token: app.use(wildcard(getContext));
This makes sense for me but I am note sure where I have to set my token in the wildcard client to send it with each endpoint query. Can you help here ? Thx!!

Hey!
You can achieve that with getContext:
const express = require('express')
const wildcard = require('@wildcard-api/server/express')

const app = express()

// We install the Wildcard middleware
app.use(wildcard(getContext))

// `getContext` is called on every API request. It defines the `context` object.
// `req` is Express' request object
async function getContext(req) {

  const authToken = req.get("Authorization")
  let user = authToken ? await tradeTokenForUser(authToken) : null

  const context = {user}
  return context
}

You can think of the context object as bridge between Express and your Wildcard functions.
The getContext function is called on every API request and you can use it to provide any information you want to your Wildcard functions.
Does that make sense?

Hi! Sry, I didn't express myself well.
I got the backend piece working but struggle to attach my authorization header to all endpoint requests.
Basically, I am looking for a way to attach the token to wildcard to make all requests have that token as an Authorization header without having to manually attach it to every request in the action.
I haven't worked with RPC before. So it could be that I am missing an option that can do this for me.  With axios I used an interceptor to attach it to all requests:
axios.interceptors.request.use(function (config) {
    const token = store.getState().session.token;
    config.headers.Authorization =  token;

    return config;
});

In graphql I added it to the httpLink
const httpLink = createHttpLink({ uri: 'http://localhost:3000/graphql' })

const authLink = setContext((_, { headers }) => {
  const token = Cookies.get('token')

  return {
    headers: {
      ...headers,
      authorization: `Bearer ${token}`
    }
  }
})

const client = new ApolloClient({
  link: authLink.concat(httpLink),
  cache: new InMemoryCache()
})

I am looking to do something similar with wildcard.
Here is a small app I created to make it more visual for you:
Backend:
endpoints.js
const {endpoints} = require('@wildcard-api/server');

endpoints.me = function() {
  return this.user
};

endpoints.login = function(name) {
    const jwt = require('jsonwebtoken');
    const jwtSecret = "mysuperdupersecret";
    const token = jwt.sign({ name}, jwtSecret, { expiresIn: 60 }) // 1 min token

    return token
};

index.js
const express = require('express')
const wildcard = require('@wildcard-api/server/express')

const jwt = require('jsonwebtoken');
const jwtSecret = "mysuperdupersecret";

const app = express()
const port = 8000

app.use((req, res, next) => {
    res.setHeader("Access-Control-Allow-Origin", "*")
    res.setHeader("Access-Control-Allow-Methods", "POST,GET,OPTIONS")
    res.setHeader("Access-Control-Allow-Headers", "Content-Type, Authorization")
    if (req.method === "OPTIONS") {
      return res.sendStatus(200)
    }
    next()
  })

  app.use(wildcard(getContext))
  app.listen(port)

async function getContext(req) {

  const authToken = req.get("Authorization")
  let user

  if (authToken) {
    user = jwt.verify(token, jwtSecret);
  }

  return  {user}
}

frontend
wildcard.js
import wildcardClient, { endpoints } from "@wildcard-api/client";
wildcardClient.serverUrl = "http://localhost:8000";

export default { wildcardClient, endpoints };

index.js
import React, { useState, useEffect } from 'react';
import { Cookies } from 'react-cookie';
import {endpoints} from '../wildcard';

const cookies = new Cookies();

export default function Index() {
  const [user, setUser] = useState(0);

  const loadUser = async () => {
    setUser(await endpoints.me() ||  'Guest')
  }

  useEffect(() => {
    loadUser()
  }, []);

  const login = async () => {
    const token = await<EMAIL_ADDRESS>password:'password'})

    if (token) {
      cookies.set('token', token);
      loadUser()
    }
  }

  return (

Hello {user}

      <button onClick={login}>Login</button>

  );
}

Hope this makes more sense to you

Wildcard doesn't have any utility to set cookies.
To process login and logout you should bypass Wildcard and use Express directly.
Or you can set cookies in the browser, which is what you are doing, but this is unusual and I'd stick to the more traditional approach of using Express directly, unless you have a good reason to deviate from the traditional approach.
Once the cookies are set you use getContext to bridge over to Wildcard.
As usual, let me know if that makes sense :-).

Btw. I'm exploring ways for allowing Wildcard users to modify the context object in a protocol-agnostic and secure way.
const {endpoints, setPrivateKey} = require('@wildcard-api/server');

endpoints.login = function(params) {
  const user = service.login(params)

  if (user) {
    // We modify the context!
    this.user = user;
  }
};

endpoints.getAllTrelloCards = function() {
  const {user} = this;
  // SQL/ORM query to get all user's Trello cards
};

// The context is secure.
// Which means you can do `this.user = {username: 'brillout'};` and
// safely assume that the user is indeed `brillout`.
// (That is, the client cannot modify the context.)
// To secure the context, Wildcard uses a private key that you provide.
setPrivateKey('AvnrrbAZ7pxH6P0s38');

// That's it!
//  - No need for `getContext`.
//  - Entirely transport protocol agnostic. No need to fiddle around with
//    HTTP headers, cookies, or localStorage.
//  - Wildcard automatically makes your context secure.
//  - Cookies are going to be deprecated soon, this is future-proof.

What do you think?

Ok, I only know the REST and GraphQl way with attaching the JWT token as an Authorization Header to each request.
Especially with SSR like Next.js / Nuxt.js; That is why I am sticking to JWT.
I can't think of any other solution dealing with authorizations but if you find one I am happy to test it out :-)

I'll show you once I've implemented the new context design. I believe you'll love it ;-).

Let me know if you are stuck btw. Your solution of setting the cookie on the browser side works for you now, correct?

kind of. I am just sending the auth token as first parameter to all of my endpoints. It's a step backwards compared to REST/GraphQL, but it works for now.
Let's wait what you come up with. Still can't really imagine it :-D

Hey @brillout ,
Following up on our twitter conversation I created this snippet. It's of course not production ready and also my first steps with React.. So please bare with me :-D
But I feel it shows what I am doing.
I am always sending the token as first argument to all my endpoints. Before running the endpoint on the server I call an intercepter to remove the token argument and forward the rest to the endpoint.
If you have a better idea to do authentication with node.js and next.js, let me know.
Important
you have to add this line to the RunEndpoint function in the WildcardApi file in the node_modules repo.. I know it is super evil... again, just for showing.
[endpointName, endpointArgs, context, isDirectCall] =  await require('../../../context')(endpointName, endpointArgs, context, isDirectCall)
https://github.com/johnson544/doorman-react
https://github.com/johnson544/doorman-node

I will have a look at it tmrw :)

I had a look at it and I'll reply tmrw ;-).

In general authorization cookies should be handled only by the server.
In your case it is only Express (or whatever server library Next.js uses) that should modify and read the cookie. You don't have to do this yourself and you can use a library instead, there are many Express auth libraries. Social login libraries usually handle this for you.
Do you really need SSR? If you don't need SSR then I'd recommend to not use Next.js but Parcel instead.
If you do need SSR then you'll need to re-bind the cookie header while doing SSR, see https://github.com/reframejs/wildcard-api/blob/master/docs/ssr-auth.md#ssr--authentication
Does that point you to the right direction?

Wildcard doesn't have any utility to set cookies.
To process login and logout you should bypass Wildcard and use Express directly.

I'm sharing a helper function via getContext that sets a cookie. What do you think of this approach? @brillout

Hi @michie1,
Yes exactly, getContext is not meant to be used to manipulate the response.
Auth is usually done outside of Wildcard and is done by the server framework instead. For example:
const express = require("express");
const cookieParser = require("cookie-parser");
const computeJwtToken = require('./path/to/computeJwtToken');

const app = express();
app.use(cookieParser());

app.get("/login", (req, res) => {

  const options = {
    maxAge: 7 * 24 * 60 * 60 * 1000, // Expires after 7 days
    httpOnly: true, // The cookie is only accessible by the web server
  };
  const jwt = computeJwtToken(req);;
  res.cookie("Authorization", jwt, options);

  res.send("login successfull");
});

You can then read the Authorization cookie in Wildcard's getContext.
I've plans to implement a new API setContext which I'll implement if the demand for it continuous to increase.
In the meantime if misusing getContext to manipulate the response works for you, then go for it. As far as I can see, there shouldn't be any problem.
Let me know if you have other questions!

Some news about this at https://github.com/reframejs/wildcard-api/issues/59.


LORA fine tune on dataset GEM/viggo

🚀 The feature, motivation and pitch
I dont think it is a bug, so i put the issue here.
how should I to fine tune on a dataset like GEM/viggo
torchrun --nnodes 1 --nproc_per_node 4  ./llama_finetuning.py --enable_fsdp --model_name NousResearch/Llama-2-7b-hf --use_peft --peft_method lora --output_dir PathFT --dataset alpaca_dataset --batch_size_training 8

data is https://huggingface.co/googcheng/7b-viggo/raw/main/file.json
my config
{
  "auto_mapping": null,
  "base_model_name_or_path": "NousResearch/Llama-2-7b-hf",
  "bias": "none",
  "fan_in_fan_out": false,
  "inference_mode": true,
  "init_lora_weights": true,
  "layers_pattern": null,
  "layers_to_transform": null,
  "lora_alpha": 32,
  "lora_dropout": 0.05,
  "modules_to_save": null,
  "peft_type": "LORA",
  "r": 8,
  "revision": null,
  "target_modules": [
    "q_proj",
    "v_proj"
  ],
  "task_type": "CAUSAL_LM"
}

but it can't inference the target formatted output data.
standard full parameter fine-tuning can make it link
SHOULD I need to add a command like "covert data to attributes" to instruction field of JSON ?
Alternatives
No response
Additional context
No response

@goog as this is a new dataset for this repo, you can follow this example and this doc to add your new dataset, also if interested feel free to make it a PR if you like to add the dataset here as well.

Hi! It seems that a solution has been provided to the issue and there has not been a follow up conversation for a long time. I will close this issue for now and feel free to reopen it if you have any questions!


Bounded Dogleg

I'm interested making a bounded Dogleg solver by wrapping the existing Dogleg, such that it supports the lower and upper arguments to maintain consistency with the interface for the Newton solver. Could you provide a couple of pointers on how to implement this, and whether you prefer I implement it into Optimistix directly or just my own code for the time being?

I think for this I'll defer to @johannahaffner who has been thinking about bounded solvers recently :)

Hi Dan,
I actually already implemented this for the Gauss-Newton solvers - bounds may be passed as options and y is then clipped to these. I've been using this for chemical reaction networks, to clip the search space to a region in which the objective is convex. I can rebase, throw in a test for the tree-clipping and push this to my own fork of optimistix, then you can use it right away.
(@patrick-kidger this could be a PR, but since the interface might change soon I would not opt for that.)
We're indeed working on constrained optimisation in optimistix, and this will include constrained trust-region solvers too, although I didn't get to that part yet (planning to change that in the next two weeks).

You can try https://github.com/johannahaffner/optimistix/tree/gauss-newton-clip.
For now I'm not checking that the first guess y0 is inside the bounded region, but since every step is clipped all others should be.

This is fantastic, thanks for sharing!


Using mocks in unit tests

Mocks in unit tests
Unit tests are aimed at testing small isolated components of code that can be predictably executed any number of times to yield the same outcome.
Mocks provide a convenient mechanism to circumvent interaction with methods that are outside the scope of your unit tests. This will let you focus on testing the code that is in scope for your unit tests and not worry about external classes/libraries/dependencies.
Having said that, code should be testable without using any of the mocking libraries like Mockito, JMockit etc. "Fakes rather than Mocks". A fake object is simply a replacement for the real object. While a real object may have actual business logic, a fake object may simply return canned responses.
public interface ConfigurationService {
        Configuration readConfiguration(String configurationName);
}

public class AzureConfigurationService implements ConfigurationService {
    Configuration readConfiguration(String configurationName) {
        // read configuration from Azure configuration service
    }
}

public class MyApplication {
    private final ConfigurationService configurationService;
    public MyApplication(ConfigurationService configurationService) {
        this.configurationService = configurationService;
    }

    public String getConfigurationOwner () {
        Configuration configuration = configurationService.readConfiguration("configName");
        return configuration.getOwner();
    }
}

public class Test {
    @Test
    public void testMyApplication() {
        // unit test won't work as it requires connection to azure services
        MyApplication myApplication = new MyApplication(new AzureConfigurationService());
        assertEquals("foo", myApplication.getConfigurationOwner());
    }
}

Instead, you could inject a fake dependency
public class FakeConfigurationService implements ConfigurationService {
    public Configuration readConfiguration(String configurationName) {
        Configuration configuration = new Configuration();
        configuration.setOwner("foo");
        return configuration;
    }
}

public class Test {
    @Test
    public void testMyApplication() {
        // unit test will now work as the fake object requires no network connection
        MyApplication myApplication = new MyApplication(new FakeConfigurationService());
        assertEquals("foo", myApplication.getConfigurationOwner());
    }
}

However, mocking libraries help in writing your tests faster. In the previous example, a fake implementation had to be created to write unit tests. Mocks can make this faster by letting you mock the behavior of real object without writing lot of code for faking dependencies.
Good candidates for using a mock:

Dependencies that require network connection - Any dependency that goes over the wire like database access, REST service calls etc.
File System - Methods/classes that require file operations. This should be treated similar to a dependency that requires network connection. Consider the case that the file may reside on a NFS.
Third-party library classes - mock these judiciously. Your code depends on these libraries and it's best to not mock them unless they are libraries that internally use file or network operations. Ideally, such libraries should be wrapped behind an interface you own and the dependency should be contained.
Date and time tests - Tests that rely on date can be mocked to return dates in future/past. Some functionalities require execution at certain intervals of time. Injecting a mock clock object with tickers that can simulate 100 clock ticks within a second is useful rather than letting the unit test run for 100 seconds.

Don't mock
Static methods - Libraries like PowerMock allow mocking static methods but it comes at a cost. It meddles with the default classloader and can lead to inconsistent behavior in tests and production runtime. Also, some of these libraries interfere with test coverage instrumentation and result in incorrect test coverage reports.
Solution: If you need to mock a static method, first evaluate if there's a way to refactor your code and eliminate the need for static method. If that's not possible, then consider isolating the static method access by wrapping it in a method that can be easily mocked.
public class ClassToTest {
    public String methodToTest() {
        String retVal = Util.notTestFriendlyStaticMethod();
        return process(retVal);
    }
}

Consider doing
public class ClassToTest {
	private UtilWrapper utilWrapper;

    public ClassToTest(UtilWrapper utilWrapper) {
        this.utilWrapper = utilWrapper;
    }

    public String methodToTest() {
        String retVal = utilWrapper.callStaticUtilMethod();
        return process(retVal);
    }
}

public class Test {
    @Test
    public void testMethod() {
        UtilWrapper utilWrapper = Mockito.mock(UtilWrapper.class);
        when(utilWrapper.callStaticUtilMethod()).thenReturn("ExpectedString");
        ClassToTest classToTest = new ClassToTest(utilWrapper);
        assertEquals("expectedstring", classToTest.methodToTest());
    }
}

Mocking Final classes
If you are using Mockito, it requires additional configuration to be able to mock final classes as described in Mockito 2 documentation.
Note that most of (if not all) the client classes provided in Azure SDK are final. These clients use the network to communicate with Azure services.
As a consumer of Azure SDK client libraries, your application is taking a dependency on 3rd party library that making network calls. In such scenarios, it's a good idea for your application to abstract out the dependency and hide it behind your own interface. This will contain the scope of dependency just within the implementation of the interface and your application is only using the interface you have defined. This allows your application to switch between different implementations of the interface without having to make changes to your application. The added benefit is that now your unit tests can either use a fake implementation as shown in the example at the top of this page or use a mock without requiring special configuration to enable mocking of final classes.

@mssfang Could this be closed? I've seen mocks being used in the AppConfiguration tests from PRs I've reviewed.

Thanks for checking this up. Close this issue now.

I honestly I am not convinced of the approach of marking Azure SDK classes with final nor providing an interfaces for these classes. Sure, having a wrapper with fake dependency works many times, not only with Azure SDK, but it should be done with every third party sdk. However, there are instances, that fake implementation (not mocked) by either implementing an interface or overriding the client class to provide these to your system code. Sure, I can mock something like this here, but honestly I don't find it sustainable to do mocking like this, I rather implement a client interface and provide a fake implementation, e.g for AWS stub, you can see there was any mocking being done there, only fake implementation.

Hello @srnagar.
Let's say that I created an interface that provides azure storage blob services to my entire application.
Once this interface is created, an implementation that uses azure blob services directly to provide storage service.
Now, how unit test case for this specific implementation would be written ? As you said, most of the azure java sdk classes are final, how can I unit test this implementation alone without mocking final classes or fakes as there is no interface available.
Please pardon me if my understanding is not upto the mark. I am still learning ways of properly writing software.

Hello @srnagar.
Let's say that I created an interface that provides azure storage blob services to my entire application.
Once this interface is created, an implementation that uses azure blob services directly to provide storage service.
Now, how unit test case for this specific implementation would be written ? As you said, most of the azure java sdk classes are final, how can I unit test this implementation alone without mocking final classes or fakes as there is no interface available.
Please pardon me if my understanding is not upto the mark. I am still learning ways of properly writing software.

I'm unable to easily mockito my azure classes still.
With 12.14.0-beta.1 it seems to be still public final class BlobServiceClient ... :|


Whole array destruction

Original issue: https://charm.cs.illinois.edu/redmine/issues/251

I want to be able to create chare arrays, have them do some work, and then completely disappear, including the underlying CkArrayMgr and CkLocMgr groups, as appropriate. This requires being able to destroy the groups themselves.

Original date: 2013-08-07 23:47:00

Implemented CkArray::ckDestroy() to do at least the basics of what one would expect from it. I think the destructor probably leaks stuff, and we need to refcount CkLocMgr and delete that when it has no more arrays bound to it. Maps are a bit stickier, in that users create those explicitly, so they would need to be destroyed explicitly.
Note that I have not addressed standard group destruction, only the particular group classes involved in chare arrays.

Original date: 2013-08-08 18:38:41

Getting rid of CkLocMgr instances will be a bit harder, as there is a connection to the load balancing machinery that needs to be cut.

Original date: 2013-08-08 20:49:27

OK, LB disconnection looks to be pretty much done - AtSync barriers are deactivated, and the object manager entry is deleted. There's still an empty table slot left behind, and might be other stuff as well, but memory usage is down substantially (per valgrind's memcheck still-reachable memory reporting). The CkLocMgr now deletes itself when the last bound array unregisters.

Original date: 2013-08-08 21:38:12

Pushed on branch array-deletion, ready for review.

Original date: 2013-08-09 20:53:11

Jonathan notes that there's potential for failure if an array element migration message is in flight when the array manager group gets destroyed. I will add a check for that case.
We concluded that it's mostly acceptable to discard those messages, rather than adding some heavy synchronization to check that all elements are properly destroyed before destroying the manager group. That means there's a slight weakness in that array classes written to do some special cleanup when each element is destroyed will have to explicitly synchronize element destruction being complete before triggering array manager destruction.

Original author: Jonathan Lifflander
Original date: 2013-08-09 21:10:13

We should note in the documentation that once 'ckDestroy' has been called on an chare array proxy, that proxy can not be used to send a message or call ckDestroy again. This will lead to undefined behavior.

Original date: 2013-08-11 23:49:37

In his review, Lukasz suggested:

I don't know whether there are other places in the code that store
array information. Two places where that may happen are array
listeners and comlib. You may want to run a benchmark where you create
and delete arrays in a loop and check that memory use does not
increase.

Original date: 2013-08-14 02:18:00

The test Lukasz suggested turned up the node-level CkArrayReductionMgr instances created for every single group, despite them going unused in non-SMP builds. Addressed in #266.

Original date: 2013-08-14 03:33:24

After testing repeated construction and deletion of chare arrays, I've squashed a bunch of residual leaks. These included various locks that only actually got allocated on SMP builds, the node-level reduction manager, and data associated with individual arrays by their map objects.
I also discovered and fixed issues #266, #267, and #268 along the way.

Original date: 2013-08-14 23:53:44

Hunted down more pathological stuff in spring cleaning: canceled Ccd triggers are not reclaimed as readily in the 'keep' mode, so transient structures like the array management groups should reset a fresh trigger each time.
I think we're finally clean in this regard. Code to be merged when mainline is clean.

Original date: 2013-08-19 20:02:42

And a maybe-final cleanup for the LB barrier trigger on element-less PEs.

Original date: 2013-08-20 23:23:15

Cleaned up the branch and merged to mainline.


how do i load new variables and alias after antidot clean

I did an antidot clean earlier and .gnupg got migrated
I ran gpg and then ~/.gnupg got created as env.sh hadn't been reloaded to set GNUPGHOME
I run source .profile (where I have $(antidot init) and nothing more was loaded, as ANTIDOT_INIT=1 was set, it wasn't loaded again.
In the end, I restarted my computer and it loaded the new variables and it was all working correctly. Is there a way I can load them without restarting my computer?

This is a bug. The init variable needs to go. I wanted to save some cpu cycles but I guess that it should source the files after every change. I will just remove that check. Thanks


How to integrate RestAPI + mobile app/SPA + social network(Google, facebook,...), using Passport

In my system, I am using Express to build rest API(api gateway) and passportJS to manage "authentication" with local, Facebook or Google.... Client can be SPA or mobile app Currently, it works well, I can get token from social network. However, my concern is "how to handle callback from Google":

If we handle on Rest API, how can redirect to client(SPA/mobile).
If we handle on client, how can to pass token to server and server validates this token is valid

Besides, passport can support to verify Google/Facebook token?
I searched a lot but cannot get the good answer. Please give me your advice and common approach to   build a API gateway.
Thanks,

Hi, I had similar issue like you. I am also beginner with Node.js, so I am not sure that this is best practice. For issue that you have described, I have used passport-facebook-token library for Facebook authentication. There are similar libraries for Google, Twitter... authentication.

Thanks, i will check and let you know soon

Any way around passport-facebook-token? since implicit grant is not best practice.
A way to know if /auth/facebook returns a successfully signed user or prompted the user with fb sign up page would be helpful, so in case it was successful u inform client (mobile app in my case) that the user is logged, if the authentication failed u can inform it as well. In the current passport authenticate fb strategy it just redirects to facebook login form.


Jump tracking for connections

Type: Feature
Priority: Normal
Requirements:

log all jumps through a connection (datetime/Pilot/Ship/mass) (mandatory)
possibility to define custom masses for ships (nice to have)

shipped since yesterday evening : commit https://github.com/pfh59/eve-whmapper/commit/20f14fc0522a060ea403d98eff8fd3b1ef95be11
Some updates are incomming

see #231